Hostname: page-component-7c8c6479df-ws8qp Total loading time: 0 Render date: 2024-03-19T06:09:47.895Z Has data issue: false hasContentIssue false

Communication of scientific uncertainty: international case studies on the development of folate and vitamin D Dietary Reference Values

Published online by Cambridge University Press:  23 September 2014

Kerry A Brown*
Affiliation:
Food Consumer Behaviour and Health Research Centre, School of Psychology, Faculty of Arts and Human Sciences, University of Surrey, Guildford GU2 7XH, Surrey, UK
Liesbeth de Wit
Affiliation:
Food Consumer Behaviour and Health Research Centre, School of Psychology, Faculty of Arts and Human Sciences, University of Surrey, Guildford GU2 7XH, Surrey, UK
Lada Timotijevic
Affiliation:
Food Consumer Behaviour and Health Research Centre, School of Psychology, Faculty of Arts and Human Sciences, University of Surrey, Guildford GU2 7XH, Surrey, UK
Anne-Mette Sonne
Affiliation:
Department of Entrepreneurship and Relationship Management, University of Southern Denmark, Kolding, Denmark
Liisa Lähteenmäki
Affiliation:
MAPP Centre, School of Business and Social Sciences, Aarhus University, Aarhus, Denmark
Noé Brito Garcia
Affiliation:
Faculty of Health Sciences, Department of Clinical Sciences, University of Las Palmas de Gran Canaria, Las Palmas, Gran Canaria, Spain
Marta Jeruszka-Bielak
Affiliation:
Department of Human Nutrition, Warsaw University of Life Sciences – SGGW (WULS-SGGW), Warsaw, Poland
Ewa Sicińska
Affiliation:
Department of Human Nutrition, Warsaw University of Life Sciences – SGGW (WULS-SGGW), Warsaw, Poland
Alana N Moore
Affiliation:
School of Exercise and Nutrition Sciences, Deakin University, Burwood, Victoria, Australia
Mark Lawrence
Affiliation:
School of Exercise and Nutrition Sciences, Deakin University, Burwood, Victoria, Australia
Monique M Raats
Affiliation:
Food Consumer Behaviour and Health Research Centre, School of Psychology, Faculty of Arts and Human Sciences, University of Surrey, Guildford GU2 7XH, Surrey, UK
*
*Corresponding author: Email Kerry.Brown@surrey.ac.uk
Rights & Permissions [Opens in a new window]

Abstract

Objective

Transparent evidence-based decision making has been promoted worldwide to engender trust in science and policy making. Yet, little attention has been given to transparency implementation. The degree of transparency (focused on how uncertain evidence was handled) during the development of folate and vitamin D Dietary Reference Values was explored in three a priori defined areas: (i) value request; (ii) evidence evaluation; and (iii) final values.

Design

Qualitative case studies (semi-structured interviews and desk research). A common protocol was used for data collection, interview thematic analysis and reporting. Results were coordinated via cross-case synthesis.

Setting

Australia and New Zealand, Netherlands, Nordic countries, Poland, Spain and UK.

Subjects

Twenty-one interviews were conducted in six case studies.

Results

Transparency of process was not universally observed across countries or areas of the recommendation setting process. Transparency practices were most commonly seen surrounding the request to develop reference values (e.g. access to risk manager/assessor problem formulation discussions) and evidence evaluation (e.g. disclosure of risk assessor data sourcing/evaluation protocols). Fewer transparency practices were observed to assist with handling uncertainty in the evidence base during the development of quantitative reference values.

Conclusions

Implementation of transparency policies may be limited by a lack of dedicated resources and best practice procedures, particularly to assist with the latter stages of reference value development. Challenges remain regarding the best practice for transparently communicating the influence of uncertain evidence on the final reference values. Resolving this issue may assist the evolution of nutrition risk assessment and better inform the recommendation setting process.

Type
Research Papers
Copyright
Copyright © The Authors 2014 

Historic public health scares (e.g. in the UK following the Phillips inquiry into variant CJD/new variant CJD, the human prion disease caused by the BSE crisis in 1996) contributed to a general commitment for transparency, openness and evidence-based (information indicating whether a belief or proposition is true or valid) decision making in food-related science and policy( Reference Lofstedt, Bouder and Wardman 1 ). This resulted in the formal separation of science and policy activities and adoption of the risk analysis framework( 2 ).

The risk analysis framework was first adapted for use in the food area following a joint FAO/WHO expert consultation in 1995( 3 ). The framework has been comprised of three interconnected activities: (i) risk assessment (scientific evaluation of health effects); (ii) risk management (policy decisions to minimise risk); and (iii) risk communication (exchange of risk information in/outside the framework). A principle has been to maintain the separation of activities while recognising the interaction between activities (e.g. separation can vary from different organisations to different groups/tasks within the same organisation responsible for different activities)( 3 ).

Central to implementation of the risk analysis framework has been the encouragement of transparency and disclosure of uncertainty in the evidence underlying scientific advice (risk assessment) or political decision making (risk management)( 2 , 4 ). The transparent handling of uncertainty may refer to ‘explicit recognition of any uncertainty either in the current state of knowledge or in the adequacy of the available data’ (e.g. accuracy/quality or degree of incomplete/quantity of data)( 5 ). Transparency and openness have been promoted to achieve greater accountability and credibility during risk analysis, improved communication between stakeholders (e.g. science, policy, industry, health practice) and safeguard the abuse of public organisation power or resources( 2 , Reference Bailar and Bailer 6 Reference Schreider, Barrow and Birchfield 8 ).

Policies of transparency can be found across various public health disciplines responsible for risk analysis activities, such as the European Food Safety Authority (EFSA), the National Institute for Health and Care Excellence (NICE) and FAO/WHO( 2 , 4 , 9 ). Implementation of these policies requires the process and outcome of a risk analysis activity to be evident and obvious, i.e. documented and understandable (transparent), as well as accessible (open)( 7 , Reference Schreider, Barrow and Birchfield 8 , 10 , Reference Turilli and Floridi 11 ). This may be related to the organisation responsible for a risk analysis activity (e.g. disclosure of organisation role and responsibility, membership, confidentiality signed agreements, declaration of interest) or the technical content of the activity (e.g. disclosure of non-published requests, methods, findings and conclusions via websites, reports and manuscripts, meeting minutes and agendas, open consultations and open meetings, release of data and study results, etc.).

A commitment to transparency and openness may seem the correct thing to do in view of due diligence towards scientific practice( Reference Schreider, Barrow and Birchfield 8 ). Nevertheless, the implementation of transparency per se may be limited or represent certain challenges( Reference Turilli and Floridi 11 ). Detailing and providing access to all elements of the risk analysis process may be challenging in terms of resource, technical, confidentiality or wider trust issues (e.g. articulating the nature of risk assessment or inherent uncertainty of scientific endeavour; detailing contradictory views during commonplace expert discussions while maintaining unanimity in the final scientific advice)( Reference Bal, Bijker and Hendriks 12 , Reference Palmer and Hardaker 13 ).

Nutrition risk assessment to derive micronutrient Dietary Reference Values (DRV) is an area where greater transparency and openness has been encouraged( Reference Dhonukshe-Rutten, Bouwman and Brown 14 ). DRV are developed in the first two steps of the risk assessment process. The identification of a nutrient-related hazard in a food/food group is established in step one (nutrient-related hazard identification). The qualitative/quantitative evaluation of adverse health effects associated with a nutrient is established in step two (nutrient-related hazard characterisation). This constitutes the development of DRV. In a full risk assessment, DRV (steps one and two) would be combined with a nutrient intake/exposure assessment (step three) to assess exposure in relation to the DRV (step four)( Reference Baines, Cunningham and Leemhuis 15 ). Once completed, the scientific advice from the risk assessment is used to inform development of recommendations in risk management. Thus, the development of DRV can be considered a precursor to the development of micronutrient recommendations. Recommendations are defined as population targets of micronutrient intake necessary for adequate growth, function and health throughout the human lifespan. These are widely used in monitoring and evaluating population intakes as well as in developing public health policies, interventions and dietary risk communications (e.g. food-based dietary guidelines to help the population meet recommended intakes)( Reference Dhonukshe-Rutten, Bouwman and Brown 14 , 16 ).

The widespread use of DRV (directly or indirectly via recommendations) belies the complexity and uncertainty with which they are developed. The first stage of DRV development has been referred to as the ‘nutrition problem formulation stage’. During problem formulation risk managers and assessors are required to establish a shared understanding of the problem and the purpose of the risk assessment( 10 ). Previously, the primary focus of DRV development was to devise intakes related to nutrient deficiency or overconsumption. More recently the remit of DRV has expanded to also encompass intakes for health benefits (including intakes to delay the onset of disease). Yet, the evidence base surrounding the role of nutrition on health has been far from certain. Hence, prevailing scientific knowledge is used to handle limitations and uncertainty in the evidence (unknown, unreliable or indefinite evidence), develop plausible assumptions and complete the DRV development process( Reference Dhonukshe-Rutten, Bouwman and Brown 14 , 17 ).

Guidelines have been produced to assist with the development of DRV (e.g. the Institute of Medicine’s Dietary Reference Intakes (DRI) for the USA and Canada; the EFSA Panel on Dietetic Products, Nutrition and Allergies’ Scientific Opinion on principles for deriving and applying DRV)( 16 , Reference Taylor 18 ). Yet, no standardised approach or agreed best practice has been used to set DRV. Furthermore, disparity has been observed in DRV developed by different national or international bodies (e.g. vitamin D DRV). The lack of agreed best practice and disparity in DRV has led to confusion among inter/national policy decision makers, health professionals, the food industry and consumers( Reference Doets, de Wit and Dhonukshe-Rutten 19 ).

An open and transparent DRV process has the potential to assist with understanding why values differ between countries by clearly detailing what, why and how decisions were made, particularly regarding the degree of uncertainty in the evidence and how this was handled or influenced the strength of the final risk assessment conclusions/resultant reference values( Reference Doets, de Wit and Dhonukshe-Rutten 19 , Reference Timotijevic, Barnett and Brown 20 ). This would inform the debate on DRV development best practice. Furthermore, increased documentation, understanding and accessibility to the DRV process could contribute to the responsible use of DRV throughout risk management and risk communication activities( 2 , Reference Dhonukshe-Rutten, Bouwman and Brown 14 ).

The European Commission-supported Network of Excellence EURRECA (EURopean micronutrient RECommendations Aligned, FOOD-CT-20006-36196, 2007–2012) was tasked with reviewing the methodologies used to derive DRV, assessing the reasons for value disparity and the potential for methodological alignment across national and international DRV development. Early EURRECA findings highlighted the importance of the nutrition problem formulation stage to ensure risk assessment activities remained achievable while also fulfilling the requirements of risk managers( Reference Timotijevic, Barnett and Brown 20 ). Further research suggested DRV disparity was unlikely to be explained by differences in concepts, definitions or defined population groups. Instead it was considered likely to be due to different interpretations or assumptions taken while accounting for uncertainty and limitations during evidence evaluation( Reference Doets, de Wit and Dhonukshe-Rutten 19 ). Finally, research identified variation in the transparency of different types of bodies in relation to how risk managers integrated DRV with other types of advice to develop recommendations( Reference Timotijevic, Barnett and Brown 20 ). The current study built upon these previous findings. Transparency and openness during the DRV risk assessment process was explored, with a particular focus on how uncertainty in the evidence base was handled. Three areas of interest, likely to demonstrate handling of uncertainty in the evidence base, were identified during DRV development: (i) the request to develop DRV; (ii) the process of DRV evidence evaluation; and (iii) the integration of evidence to develop final DRV.

Method

Design

Qualitative in-depth case studies explored the handling of uncertain evidence and the transparency of vitamin D and folate DRV development in six countries/regions: (i) Australia and New Zealand (ANZ); (ii) the Netherlands (NL); (iii) the Nordic countries (Denmark, Finland, Iceland, Norway and Sweden; NOC); (iv) Poland (PL); (v) Spain (ES)( Reference Garcia and Majem 21 ); and (vi) the United Kingdom (England, Northern Ireland, Scotland and Wales; UK).

Each country/region produced a descriptive report based upon semi-structured interviews with at least two advisory committee members together with desk research (Table 1). A common protocol was followed throughout. The interview schedule, desk research and final case study report all centred upon the transparency and handling of uncertain evidence in three predefined areas: (i) the request to develop DRV, e.g. source of request, scope of request, reasons for the request; (ii) the process of DRV evidence evaluation, e.g. selection and evaluation of evidence; and (iii) integration of evidence to develop final DRV, e.g. formation of quantitative numbers and advisory committee report.

Table 1 Country/region, number of interview samples, response rate, and references and associated updates for the folate and vitamin D DRV

DRV, Dietary Reference Values; ANZ, Australia and New Zealand; NL, Netherlands; NOC, Nordic countries (Denmark, Finland, Iceland, Norway and Sweden); PL, Poland; ES, Spain; UK, United Kingdom; NHMRC, Australian National Health and Medical Research Council; HCN, Health Council of the Netherlands (Gezondheldsraad); NNR, Nordic Council of Ministers’ Nordic Nutrition Recommendations; SENC, Spanish Society of Community Nutrition; FESNAD, Federación Española de Sociedades de Nutrición, Alimentación y Dietética; COMA, Committee on Medical Aspects of Food Policy; SACN, Scientific Advisory Committee on Nutrition.

* Recommendations under development at time of study data collection.

Transparency and openness was qualitatively judged via the availability of accessible, understandable documentation on the DRV process (e.g. downloadable/upon request meeting agendas, minutes and key discussion documents; ability to attend open meetings and consultations and documentation surrounding DRV; documented risk analysis policy, organisation role and responsibility, membership, declaration of interest; stakeholder involvement; etc.).

Vitamin D and folate were selected due to their prioritisation as micronutrients of interest in Europe( Reference Cavelaars AEJM and Dhonukshe-Rutten 22 ). Countries/regions were originally selected from Europe with a subsequent opportunity taken to also collect data in Australia and New Zealand. The countries/regions sampled represented diversity in the age and method of developing DRV as well as in geographical location, socio-cultural factors and institutional infrastructure. Data sets were collected from December 2010 to June 2011.

Interviews

Interviewees were recruited due to their involvement in the advisory committee and/or development of vitamin D or folate reference values. This included members of the advisory committee and/or members of relevant advisory committee working subgroups. The response rate ranged from 33 % (UK) to 100 % (ES and PL; Table 1). Consent was obtained from each participant, with interviews recorded and later transcribed verbatim. All data were made anonymous and held in accordance with the local data privacy laws (e.g. UK Data Protection Act)( 23 ). The semi-structured interview schedule was piloted and devised by the research team with questions focused upon the transparency and handling of uncertainty in the aforementioned three areas of DRV development.

Desk research

Sourced documents related to the development of vitamin D or folate DRV in each country/region. Keyword searches were conducted of advisory committee websites, publication websites (e.g. PubMed Central) and search engines (e.g. Google). Documents referred to during the interviews and manual searches of references elicited several additional documents. Further information was obtained in the UK case study via observation at two open advisory committee meetings (14 February 2011 and 7 June 2011). Desk research key search terms included variants of the nutrient name (e.g. folate and folic acid), variants of the word DRV (e.g. micronutrient recommendations, nutrient intake values, nutritional objectives) and/or the advisory committee name (e.g. Health Council for the Netherlands). Excluded documents were not relevant to vitamin D or folate DRV, the specific country/region, human nutrition or healthy populations or were duplications from previous searches. Desk research was led and organised by questioning the transparency and handling of uncertainty in the aforementioned three areas of DRV development.

Analysis

Initial interview and desk research analysis was conducted in the native language using template analysis and a skeleton coding structure created and modified by partners during preliminary analyses( Reference Boyatzis 24 ). Case studies consisted of an English-translated summary of interview- and/or desk research-identified themes with illustrative quotes organised by the three areas (value request, evidence evaluation and final values). Construct validity was upheld by limiting the subjective collection of data and performing traceable literature searches. Internal validity was upheld by encouraging several interpretations of the data to be considered. Multiple sources of information (triangulation between interview, desk research and DRV reports where possible) were used to cross-reference (corroborate/dispute findings). The uniform framework of case study reporting allowed further analysis where cross-case synthesis was used to describe and identify any differences or similarities across countries/regions in the transparency and handling of uncertainty( Reference Yin 25 ). One research team member conducted the cross-case synthesis. The qualitative software NVivo version 9 (2010) was used to organise case study data. A subsection of the data was cross-coded by another research team member. One fewer theme was identified by the second coder. This was subsequently dropped from further analysis. Agreement on remaining themes and sub-themes was 91%. Interpretation of cross-case analysis results was reflected back to case study authors and amended where necessary.

Data presentation

Interview quotes have been displayed in the tables using double quotation marks and italics. Verbatim quotes have been modified in publication for improved readability. Case study/desk research text has been displayed in the tables using single quotation marks. The term ‘micronutrient recommendation’ was used in the interview schedule and throughout the present study. This term could be translated and understood across the cases sampled as pertaining to either micronutrient DRV or the use of values to make recommendations.

Note that quotes have been unaltered in this respect and differences in the terminology can be seen as cases refer to their respective micronutrient recommendations, e.g. ‘The term “Nordic Nutrition Recommendations” refers to a set of dietary reference values (DRVs) for essential nutrients that includes the average requirement (AR), recommended intake (RI), upper intake level (UL), lower intake level (LI), and reference values for energy’( 26 ).

Results

Transparency and openness of the DRV process (i.e. documentation, understanding and accessibility) varied across the countries/regions (cases) and different areas of the DRV development process studied. Results have been presented below in the three areas studied: (i) the request to develop DRV; (ii) the process of DRV evidence evaluation; and (iii) the integration of evidence to develop final DRV.

Request to develop DRV

In ES, where multiple sets of DRV were developed by different individuals, teams and organisations (Spanish Society of Community Nutrition (SENC), Federación Española de Sociedades de Nutrición, Alimentación y Dietética (FESNAD), Universidad Complutense de Madrid), minimal documentation could be found to clarify details on the source, scope or reason for the request to develop DRV (meeting agendas/minutes, discussion documents, press releases, website pages, final report details, journal publications). However, the Spanish interview data were more informative and interviewees openly discussed details surrounding the request for previous DRV. In contrast, greater information (via desk research and interview data) on the request to develop DRV was available in countries/regions with an official set of DRV and a clear body tasked with their development, such as in ANZ (National Health and Medical Research Council (NHMRC)/Ministry of Health (MOH)), NL (Health Council of the Netherlands (HCN)), NOC (Nordic Council of Ministers (NCM)), PL (National Food and Nutrition Institute (Instytutu Żywności i Żywienia; IŻŻ)) and UK (Committee on Medical Aspects of Food Policy (COMA)/Scientific Advisory Committee on Nutrition (SACN); Tables 1 and 2).

Table 2 Request to develop DRV

DRV, Dietary Reference Values; ES, Spain; SACN, Scientific Advisory Committee on Nutrition; UK, United Kingdom; PL, Poland; MHWS, Ministry of Health, Welfare and Sport; NL, Netherlands.

“Italics and double quotation marks denote interview quotes”; ‘single quotation marks denote case study/desk research text’.

Regarding the source of the request to develop DRV, Spanish interviewees cited the majority of activity was undertaken by independent and academic institutions. On the other hand, in the five cases of ANZ, NL, NOC, PL and UK the requests were predominantly sourced from the government or within the advisory committee as a set programme of work.

Across all six cases the reasons provided for the request to develop DRV ranged from acknowledged discrepancy between local DRV and those of other countries, DRV age, the emergence of new data, a set time cycle for the programme of work, information provision to assist policy option selection and triggers from monitoring practices, such as clinical health outcomes or markers of chronic disease risk (e.g. neural tube defects, LDL). However, details on the final trigger to initiate a request for particular nutrient DRV to be reviewed or developed were not always transparent, i.e. the influence of lobbying (professional, academic, commercial or special interest group) on initiating or suppressing any development of DRV. Similarly, the criteria for judging when DRV were ‘too old’, when there was ‘sufficient’ new evidence to initiate renewing DRV or at what point disease incidence constituted a public health issue to address were not always transparently documented.

The scope of request differed between cases. In the ANZ, PL and ES cases the scope of the request centred upon the utilisation and adaptation of international DRV to each respective country/region (predominantly the US/Canadian DRV in ANZ and PL; various international DRV in ES). This was clearly detailed in ANZ. The NHMRC was asked to assess the body of evidence used to establish the US/Canadian DRV (and any relevant literature that had subsequently been released) and provide an opinion on one of four courses of action: whether they should adopt, adopt with minor changes, adopt with substantial changes or reject the US/Canadian DRV in ANZ.

The remaining three cases (NL, NOC and UK) all referred to the importance of clarifying the scope of the request and iterative interactions between government representatives and advisory committee members during the nutrition problem formulation (e.g. to demarcate risk assessment/risk management activities or develop values for deficiency/adequacy/optimal/toxicity). Problem formulation was deemed necessary to ensure the request would be both appropriate and achievable (e.g. scientific limits of knowledge and resource constraints such as expertise, finance, time). Transparency appeared to be the greatest regarding the current UK vitamin D DRV development process. Here, the public was allowed to attend an open meeting where the scope of the request was discussed (also known as ‘terms of reference’ in the UK) and online access was made available for downloading detailed meeting minutes and discussion documents surrounding request clarification.

Process of DRV evidence evaluation

Several discussions were undertaken on how to evaluate and interpret evidence that contained methodological and theoretical uncertainties (e.g. folate assay method, folate equivalents, uncertainty factors, bioavailability factors, vulnerable population groups, mechanisms of action, multiple nutrient interactions). These discussions were present in the cases which adopted/adapted existing international DRV as well as those which developed their own DRV (Table 3). For example, the ANZ, PL and ES cases made reference to the evidence evaluation previously conducted in the original DRV as well as the additional evidence evaluation required to ensure adapted DRV were up-to-date and relevant to their respective country/region.

Table 3 Process of DRV evidence evaluation

DRV, Dietary Reference Values; NHMRC, Australian National Health and Medical Research Council; ANZ, Australia and New Zealand; ES, Spain; CBO, Dutch Institute for Healthcare Improvement; SIGN, Scottish Intercollegiate Guidelines Network; NL, Netherlands; NNR5, Nordic Council of Ministers’ Nordic Nutrition Recommendations, 5th edition; NOC, Nordic countries (Denmark, Finland, Iceland, Norway and Sweden); SACN, Scientific Advisory Committee on Nutrition; UK, United Kingdom.

“Italics and double quotation marks denote interview quotes”; ‘single quotation marks denote case study/desk research text’.

To increase the transparency and scientific rigour of the evidence evaluation process, four cases (ANZ, NL, NOC, UK; Table 4) used protocols to guide evidence evaluation, address uncertainty in the data and limit interpretation bias. These evidence evaluation protocols differed slightly. Yet, all four cases provided guidance on the systematic reviewing of literature (including meta-analyses), the assessment of individual study quality/risk of bias and accounting for uncertainty when weighting or deciding the strength of the evidence base.

Table 4 Protocols used to assess study quality/risk of bias across cases (PL/ES unknown)

PL, Poland; ES, Spain; ANZ, Australia and New Zealand; NHMRC, Australian National Health and Medical Research Council; NL, Netherlands; CBO, Dutch Institute for Healthcare Improvement; SIGN, Scottish Intercollegiate Guidelines Network; NOC, Nordic countries (Denmark, Finland, Iceland, Norway and Sweden); NNR5, Nordic Council of Ministers’ Nordic Nutrition Recommendations, 5th edition; UK, United Kingdom; SACN, Scientific Advisory Committee on Nutrition; RCT, randomised controlled trial.

ANZ, NL and NOC used a series of checklists to assess study quality and a number of categories to differentiate between results from different study designs. The UK also utilised a series of checklists but discouraged the use of numerical grading. It was clearly acknowledged that a degree of subjective judgement was still required with the application of these protocols, especially when moving from the quality assessment of each study to collating the strength of the total evidence. Nevertheless, the following of a protocol for evidence evaluation and reporting of results was seen by these four cases as important steps towards a standardised approach to the evidence evaluation decision-making process.

Integration of evidence to develop final DRV

Interviewees across all cases acknowledged that evidence evaluation did not constitute the end of the DRV development process. Evidence evaluation findings needed to be appraised in the context of the original request to form the final DRV. Expert judgement was required to take account of the underlying certainty of the evidence, decide the strength of evidence and conclude with specific reference values. This stage of the process appeared to be less standardised with few transparent protocols available to aid the expert deliberation and consensus seeking decision-making process (Table 5).

Table 5 Integration of evidence to develop final DRV

DRV, Dietary Reference Values; NRV, Nutrient Reference Values; NHMRC, Australian National Health and Medical Research Council; ANZ, Australia and New Zealand; NNR, Nordic Council of Ministers’ Nordic Nutrition Recommendations; NOC, Nordic countries (Denmark, Finland, Iceland, Norway and Sweden); ES, Spain; NL, Netherlands.

“Italics and double quotation marks denote interview quotes”; ‘single quotation marks denote case study/desk research text’.

The four cases which employed evidence evaluation protocols (ANZ, NL, NOC, UK) appended these protocols to reports and used them to guide descriptions of the strength and degree of certainty in the evidence underlying each DRV. In addition, prior to the publication of DRV reports, the standard practices of peer review (NL, NOC, PL) and invited or open consultation (ANZ, NL, NOC, PL, UK) provided an element of transparency and credibility to the process. Nevertheless, from the interviews it was clear that considerable discussions occurred between the evidence evaluation stage and the creation of the final DRV, even in relation to the wording or terminology used to describe DRV concepts (particularly PL) or the strength of evidence/degree of uncertainty (particularly UK). However, the details from many of these discussions were rarely communicated in the final report.

Two interviewees (ES, NL) specifically mentioned that the exact discussions undertaken by those developing DRV were not commonly reported. Reasons cited for this included a concern for the confidentiality of those who developed the recommendations as well as a belief that it was unnecessary to report every aspect of discussions that occurred during the deliberations before a consensus was reached. An exception was observed in NL when a disagreement of an advisory committee member could not be resolved during the formation of the folate DRV( 27 ). Consequently, a footnote was added to the Dutch report detailing a minority position of disagreement regarding the expression of folate v. folic acid as well as folate equivalent selection.

Discussion

Transparency and openness of the DRV process was not universally adopted across countries or areas of the DRV development process studied. Implementation of transparency policies may be limited by a lack of dedicated resources and best practice procedures, particularly to assist with the latter stages of reference value development.

Findings aligned with previous research regarding transparency in risk management activities( Reference Timotijevic, Barnett and Brown 20 ). The countries with dedicated advisory committees for risk assessment activities (ANZ, NL, NOC, PL and UK) demonstrated greater transparency of the DRV process than ES, where coordinated action at a national level was in its infancy and DRV had been developed by dedicated individuals. Arguably, NL, NOC and UK appeared to display the most transparency (also potentially ANZ, although the underpinning US/Canadian DRV setting process was not studied). Similarly to the above, this could be explained by a greater legal and clarified role of the relevant advisory committees and their responsibilities (demarcation between risk assessors, risk managers and risk communicators), a higher political priority or greater dedication of resources for nutrition and a longer tradition of DRV development. Furthermore, both NOC and UK were currently updating DRV at the time of study. Thus, the nature of transparency observed may have reflected the current increased calls for transparency and been associated with the most recent micronutrient DRV setting processes.

All cases demonstrated differences in transparency (in general and specific to the handling of uncertain evidence) across the three a priori areas studied (value request, evidence evaluation and final values). Based upon these findings, the status quo for current best practice regarding transparent and open handling of uncertainty in nutrition risk assessment has been discussed below.

Regarding the request to develop DRV, transparency was increased via documentation and accessibility to the iterative discussions between risk assessors, risk managers and others in problem formulation. These discussions acknowledged the uncertainty and limitations of the evidence base available to ensure the feasibility of the risk assessment task. This transparent detailing of the source, scope and reasons for any DRV request corresponded with guidance seen in a number of international risk assessment procedural documents (e.g. the nutrition problem formation stage before nutrition risk assessment activity in CODEX Alimentarius; defining the problem in the EURRECA framework for setting micronutrient recommendations)( 10 , Reference Dhonukshe-Rutten, Bouwman and Brown 14 ).

Transparency in risk assessment evidence evaluation is designed to minimise the probability of bias( Reference Lichtenstein, Yetley and Lau 28 ). Best practice for transparent handling of uncertainty in the cases studied referred to the documented protocols designed to guide and standardise the process of sourcing and interpreting evidence. Bias was reduced by employing an independent body to conduct the review and reproducibility was increased by employing standardised systematic literature review processes. Difficulties were still seen regarding the handling of uncertainty during evidence evaluation and communicating how this influenced the quantification/qualification of risk (e.g. the merits of grading different types of evidence). In addition, whether restricting a review to evidence from randomised controlled trials or meta-analyses limited the scope of the review or the utility of the final advice/reference values. However, the constant updating and disclosure of evidence evaluation procedures in the most recent cases reflected an ability to detail the overall strength of evidence reviewed.

Few protocols were available to assist with the articulation of, or guide the transition from, evidence evaluation to the integration of evidence and development of quantitative values/final DRV. This has been highlighted previously in the health-care area, where difficulties have been observed translating evidence evaluation into scientific advice for use in recommendation/guideline development( Reference Sackett, Rosenberg and Gray 29 , Reference Michie, Berentson-Shaw and Pilling 30 ). These findings support the requirement for further initiatives such as the EFSA consultation exercise in 2013 on how to conduct consistent procedures for expert elicitation during risk assessment( 31 ).

Cases demonstrated that consensus seeking was an essential stage of the DRV development process, designed to increase validity and reliability of decisions rather than relying upon an individual’s judgement. However, it was difficult to identify the best practice for transparently developing quantitative DRV; values that require a certainty not always apparent in the underlying data. The production of a transparent DRV report documenting the complexity of evidence-based expert decision making, the strength of advice and the areas of uncertainty remained a challenge. This had the potential to pose subsequent problems for the interpretation of DRV by those outside the advisory committee (e.g. responsible for setting micronutrient recommendations/relevant toxicology risk assessment activities)( Reference Innvaer 32 Reference Dobrow, Goel and Lemieux-Charles 34 ).

The present study was based upon qualitative case studies. Therefore, the selection of micronutrients, countries and the three a priori defined areas of study may have limited external validity and the degree to which these findings can be transferred outside the sample studied. The authors recognise the study would have benefited from including analysis of other countries that develop micronutrient DRV, e.g. the USA/Canada. The exploratory nature of the present study justified the use of a qualitative design and steps were taken to limit biased interpretation. A common protocol and the three a priori defined areas of study were employed to maximise study rigour via clarity of the research goal and the consistent method of data collection, analysis and reporting. This also enabled the combination of data across countries. Construct validity was upheld by limiting the subjective collection of data, regular monthly research team meetings and performing traceable literature searches. Internal validity was upheld by encouraging several interpretations of the data to be considered. To maintain the cultural context and authenticity of the data, the majority of qualitative interpretation was conducted in the native language and information from interviews, observation or desk research provided the ability to cross-reference findings. The case studies (data collection, analysis and reporting) were led by the three a priori defined areas and did not represent the totality of the DRV development process. Nevertheless, they have provided an insight into the nature of transparency, in particular regarding the handling of uncertain data, available in an evidence-based decision-making process across multiple countries/regions.

Debate shall continue regarding the best practice for nutrition risk assessment, particularly handling uncertainty in the evidence surrounding nutrition, diet, lifestyle and health and rating the overall evidence underpinning DRV( Reference Russell 35 ). Future discussions may focus on adaptations to the risk analysis framework to facilitate increasing requirements for nutrition risk–benefit assessment in the development of DRV( 26 , Reference Tijhuis, de Jong and Pohjola 36 ). Furthermore, how to meet the challenge of incorporating developing evidence bases into risk assessment, such as those related to individual differences (e.g. metabolomics), non-randomised controlled trial study designs or whole-diet approaches (e.g. epidemiology)( Reference Baines, Cunningham and Leemhuis 15 ).

It may not be possible or necessary for all countries/regions to follow an aligned DRV risk assessment procedure nor for all reference values to be identical. However, transparency, as well as increasing the accountability and credibility of DRV development, can facilitate the sharing of best practice to inform the evolution of nutrition risk assessment. Therefore, the recent transparency initiatives from the EFSA, such as proposals to promote public access to risk assessment technical data, are to be welcomed( 37 ).

Conclusion

Implementation of transparency policies may be limited by a lack of dedicated resources and best practice procedures, particularly to assist with the latter stages of reference value development. Challenges remain regarding the best practice for transparently communicating the influence of uncertain evidence on the final reference values. Resolving this issue may assist the evolution of nutrition risk assessment and better inform the recommendation setting process.

Acknowledgements

Acknowledgements: The authors would like to acknowledge the contributions made to study design, data collection and analysis by Wojciech Roszkowski (WULS-SGGW). Financial support: The work herein was carried out within the EURRECA Network of Excellence (http://www.eurreca.org), financially supported by the Commission of the European Communities, Specific Research Technology and Development (RTD) Programme Quality of Life and Management of Living Resources within the Sixth Framework Programme, contract no. 036196. The Commission had no role in the design, analysis or writing of this article. This article does not necessarily reflect the Commission’s views or its future policy in this area. Conflict of interest: M.M.R. joined SACN in 2011, after the results in this paper were analysed. No other conflicts of interest have been declared by the authors. Authorship: The contributions of authors were as follows: K.A.B., L.d.W., L.T., A.-M.S., L.L., N.B.G., M.J.-B., E.S. and M.M.R. were responsible for the study concept and design. K.A.B., L.d.W., A.-M.S., N.B.G., M.J.-B., E.S. and A.N.M. conducted data collection, analysis and the composition of individual country summary reports. K.A.B. coordinated data collection and analysed the data across countries. K.A.B. drafted and wrote the manuscript. All authors were involved in manuscript revisions. Ethics of human subject participation: Ethical approval was not necessary for this body of work. However, this study was conducted according to the guidelines laid down in the Declaration of Helsinki. Written/verbal (witnessed and formally recorded) informed consent was obtained from all respondents, with all data made anonymous and held in accordance with the local data privacy laws.

References

1. Lofstedt, R, Bouder, F, Wardman, J et al. (2011) The changing nature of communication and regulation of risk in Europe. J Risk Res 14, 409429.Google Scholar
2. European Food Safety Authority (2009) Transparency in risk assessment – Scientific aspects. Guidance of the scientific committee on transparency in the scientific aspects of risk assessments carried out by EFSA. Part 2: General principles. EFSA J 1051, 122.Google Scholar
3. World Health Organization/Food and Agriculture Organization of the United Nations (1995) Application of Risk Analysis to Food Standards Issues. Report of the Joint FAO/WHO Expert Consultation, Geneva, Switzerland, 13–17 March 1995. Geneva: WHO.Google Scholar
4. Food and Agriculture Organization of the United Nations/World Health Organization (2007) Framework for the Provision of Scientific Advice on Food Safety and Nutrition. Rome: FAO.Google Scholar
5. World Health Organization/Food and Agriculture Organization of the United Nations (2004) Provision of Scientific Advice to Codex and Member Countries. Report of a Joint FAO/WHO Workshop, WHO Headquarters, Geneva, Switzerland, 27–29 January 2004. Geneva: WHO.Google Scholar
6. Bailar, JC 3rd & Bailer, AJ (1999) Risk assessment – the mother of all uncertainties – disciplinary perspectives on uncertainty in risk assessment. Ann N Y Acad Sci 895, 273285.CrossRefGoogle ScholarPubMed
7. European Commission (2006) Green Paper. European Transparency Initiative. COM(2006) 194 final, 3rd May. Brussels: EC.Google Scholar
8. Schreider, J, Barrow, C, Birchfield, N et al. (2010) Enhancing the credibility of decisions based on scientific conclusions: transparency is imperative. Toxicol Sci 116, 57.CrossRefGoogle ScholarPubMed
9. National Institute for Health and Care Excellence (2012) Process and Methods Guides . The Guidelines Manual. London: NICE.Google Scholar
10. Codex Alimentarius Commission (2010) Procedural Manual, 19th ed. Rome: FAO.Google Scholar
11. Turilli, M & Floridi, L (2009) The ethics of information transparency. Ethics Inform Technol 11, 105112.CrossRefGoogle Scholar
12. Bal, R, Bijker, WE & Hendriks, R (2004) Democratisation of scientific advice. BMJ 329, 13391341.Google Scholar
13. Palmer, TN & Hardaker, PJ (2011) Handling uncertainty in science. Philos Trans A Math Phys Eng Sci 369, 46814684.Google Scholar
14. Dhonukshe-Rutten, RAM, Bouwman, J, Brown, KA et al. (2013) EURRECA – evidence-based methodology for deriving micronutrient recommendations. Crit Rev Food Sci 53, 9991040.Google Scholar
15. Baines, J, Cunningham, J, Leemhuis, C et al. (2011) Risk assessment to underpin food regulatory decisions: an example of public health nutritional epidemiology. Nutrients 3, 164185.CrossRefGoogle ScholarPubMed
16. European Food Safety Authority, Panel on Dietetic Products, Nutrition and Allergies (2010) Scientific opinion on principles for deriving and applying dietary reference values. EFSA J 8, 1458.Google Scholar
17. National Research Council (2007) Nutritional Risk Assessment: Perspectives, Methods, and Data Challenges, Workshop Summary. Washington, DC: The National Academies Press.Google Scholar
18. Taylor, C (2008) Framework for DRI Development: Components ‘Known’ and Components ‘To Be Explored’. Background Paper. Washington, DC: Institute of Medicine of the National Academies; available at http://www.iom.edu/~/media/Files/Activity%20Files/Nutrition/DRIWS/Background%20Paper%20on%20DRI%20Framework%202008.pdf Google Scholar
19. Doets, EL, de Wit, LS, Dhonukshe-Rutten, RAM et al. (2008) Current micronutrient recommendations in Europe: towards understanding their differences and similarities. Eur J Nutr 47, 1740.Google Scholar
20. Timotijevic, L, Barnett, J, Brown, K et al. (2011) The process of setting micronutrient recommendations: a cross-European comparison of nutrition-related scientific advisory bodies. Public Health Nutr 14, 716728.Google Scholar
21. Garcia, NB & Majem, LS (2012) Micronutrient recommendations and policies in Spain; the cases of iodine, folic acid and vitamin D. Nutr Hosp 27, 16101618.Google Scholar
22. Cavelaars AEJM, Doets EL, Dhonukshe-Rutten, RAM et al. (2010) Prioritizing micronutrients for the purpose of reviewing their requirements: a protocol developed by EURRECA. Eur J Clin Nutr 64, Suppl. 4, S19S30.CrossRefGoogle ScholarPubMed
23. UK Parliament (1998) UK Data Protection Act, Chapter 29. London: HMSO.Google Scholar
24. Boyatzis, R (1998) Transforming Qualitative Information: Thematic Analysis and Code Development. Thousand Oaks, CA: Sage Publications.Google Scholar
25. Yin, R (2003) Case Study Research: Design and Methods, 3rd ed. Thousand Oaks, CA: Sage Publications.Google Scholar
26. Nordic Council of Ministers (2014) Nordic Nutrition Recommendations (NNR) 2012. Integrating Nutrition and Physical Activity, 5th ed. Copenhagen: Nordic Council of Ministers.Google Scholar
27. Health Council of the Netherlands (2003) Dietary Reference Intakes: Vitamin B 6 , Folic Acid and Vitamin B 12 . Report no. 2003/04. The Hague: HCN.Google Scholar
28. Lichtenstein, AH, Yetley, EA & Lau, J (2008) Application of systematic review methodology to the field of nutrition. J Nutr 138, 22972306.Google Scholar
29. Sackett, DL, Rosenberg, WMC, Gray, JAM et al. (1996) Evidence based medicine: what it is and what it isn’t. BMJ 312, 7172.CrossRefGoogle ScholarPubMed
30. Michie, S, Berentson-Shaw, J, Pilling, S et al. (2007) Turning evidence into recommendations: protocol for a study of guideline development groups. Implement Sci 2, 29.Google Scholar
31. European Food Safety Authority (2014) Guidance on expert knowledge elicitation in food and feed safety risk assessment. EFSA J 12, 3734.Google Scholar
32. Innvaer, S (2009) The use of evidence in public governmental reports on health policy: an analysis of 17 Norwegian official reports (NOU). BMC Health Serv Res 9, 177.CrossRefGoogle ScholarPubMed
33. Bayer, R, Johns, DM & Galea, S (2012) Salt and public health: contested science and the challenge of evidence-based decision making. Health Aff (Millwood) 31, 27382746.CrossRefGoogle Scholar
34. Dobrow, MJ, Goel, V, Lemieux-Charles, L et al. (2006) The impact of context on evidence utilization: a framework for expert groups developing health policy recommendations. Soc Sci Med 63, 18111824.Google Scholar
35. Russell, RM (2008) Current framework for DRI development: what are the pros and cons? Nutr Rev 66, 455458.Google Scholar
36. Tijhuis, MJ, de Jong, N, Pohjola, MV et al. (2012) State of the art in benefit–risk analysis: food and nutrition. Food Chem Toxicol 50, 525.CrossRefGoogle ScholarPubMed
37. European Food Safety Authority (2013) EFSA promotes public access to data in transparency initiative. Press release, 14 Jan 2013. http://www.efsa.europa.eu/en/press/news/130114.htm (accessed September 2014).Google Scholar
38. National Health and Medical Research Council (2006) Nutrient Reference Values for Australia and New Zealand Including Recommended Dietary Intakes. Canberra: NHMRC.Google Scholar
39. Health Council of the Netherlands (2008) Towards An Optimal Use of Folic Acid. Report no. 2008/02E. The Hague: HCN.Google Scholar
40. Health Council of the Netherlands (2000) Dietary Reference Intakes: Calcium, Vitamin D, Thiamin, Riboflavin, Niacin, Pantothenic Acid, and Biotin. Report no. 2000/12. The Hague: HCN.Google Scholar
41. Health Council of the Netherlands (2008) Towards An Adequate Intake of Vitamin D. Report no. 2008/15E. The Hague: HCN.Google Scholar
42. Nordic Council of Ministers (2004) Nordic Nutrition Recommendations 2004. Integrating Nutrition and Physical Activity. Copenhagen: Nordic Council of Ministers.Google Scholar
43. Jarosz, M & Bułhak-Jachymczyk, B (2008) Human Nutrition Standards. Fundamentals of Obesity and Prevention of Non-Communicable Diseases. Warsaw: PZWL.Google Scholar
44. Moreiras, O, Carbajal, Á, Cabrera, L et al. (2011) Tablas de Composición de Alimentos, 15ª ed. Madrid: Ediciones Pirámide S.A.Google Scholar
45. Ortega, RM, Navia, B, López-Sobaler, AM et al. (2011) Ingestas Diarias Recomendadas de Energía y Nutrientes para Población Española. Madrid: Departamento de Nutrición, Universidad Complutense.Google Scholar
46. Serra Majem, L, Aranceta, J on behalf of the SENC working group on Nutritional Objectives for the Spanish Population (2001) Nutritional objectives for the Spanish population: consensus from the Spanish Society of Community Nutrition. Public Health Nutr 4, 14091413.Google Scholar
47. Martínez, A, Cuervo, M, Baladia, E et al. (2010) Ingestas Dietéticas de Referencia (IDR) para la Población Española. Madrid: Federación Española de Sociedades de Nutrición, Alimentación y Dietética.Google Scholar
48. Department of Health, Committee on Medical Aspects of Food Policy (1991) Dietary Reference Values (DRVs) for Food Energy and Nutrients for the UK: Report of the Panel on DRVs of the Committee on Medical Aspects of Food Policy. Report on Health and Social Subjects no. 41. London: HMSO.Google Scholar
49. Scientific Advisory Committee on Nutrition (2006) Folate and Disease Prevention. London: TSO.Google Scholar
50. Scientific Advisory Committee on Nutrition (2007) Update on Vitamin D. Position Statement by the Scientific Advisory Committee on Nutrition. London: TSO.Google Scholar
51. National Health and Medical Research Council (2000) How to Use the Evidence: Assessment and Application of Scientific Evidence. Canberra: NHMRC.Google Scholar
52. Dutch Institute for Healthcare Improvement (2007) Evidence-Based Guideline Development. Guide to Working Group Members. Utrecht: CBO.Google Scholar
53. Scottish Intercollegiate Guidelines Network (2007) SIGN 50: A Guideline Developers’ Handbook. Edinburgh: SIGN.Google Scholar
54. NNR5 Working Group (2011) A Guide for Conducting Systematic Literature Reviews for the 5th edition of the Nordic Nutrition Recommendations. Copenhagen: Nordic Council of Ministers.Google Scholar
55. Scientific Advisory Committee on Nutrition (2008) SACN Framework for Evaluation of Evidence. London: SACN.Google Scholar
Figure 0

Table 1 Country/region, number of interview samples, response rate, and references and associated updates for the folate and vitamin D DRV

Figure 1

Table 2 Request to develop DRV

Figure 2

Table 3 Process of DRV evidence evaluation

Figure 3

Table 4 Protocols used to assess study quality/risk of bias across cases (PL/ES unknown)

Figure 4

Table 5 Integration of evidence to develop final DRV