Hostname: page-component-77c89778f8-gvh9x Total loading time: 0 Render date: 2024-07-16T13:28:28.308Z Has data issue: false hasContentIssue false
  • English
  • Français

Towards a Regulatory Cycle? The Use of Evaluative Information in Impact Assessments and Ex–post Evaluations in the European Union

Published online by Cambridge University Press:  20 January 2017

Thomas van Golen  and
Stijn van Voorst
Thomas van Golen
Affiliation:
Tilburg University Law School
Stijn van Voorst
Affiliation:
Tilburg University Law School
Get access Rights & Permissions [Opens in a new window]

Extract

As a part of its Better Regulation agenda, the European Commission increasingly stresses the link between different types of regulatory evaluations. Predictions made by Impact Assessments (IAs) could be verified during ex–post legislative evaluations, while ex–post evaluations in turn could recommend amendments to be studied in future IAs. This article combines a dataset of 309 ex–post legislative evaluations (2000-2014) and a dataset of 225 IAs of legislative updates (2003-2014) to show how many ex–post evaluations of the Commission use IAs and vice versa. This way, it explores if the Commission's rhetoric of a ‘regulatory cycle’ holds up in practice. Building on the literature of evaluation use, we formulate the hypotheses that the timeliness, quality and focus of the IAs and evaluations are key explanations for use. Our results show that so far only ten ex–post evaluations have used IAs of EU legislation, while thirty three IAs have used ex–post legislative evaluations. Using Fuzzy set Qualitative Comparative Analysis, we find that timeliness is a necessary condition of the use of ex–post evaluations by IAs, suggesting that for the regulatory cycle to function properly, it is crucial to complete an ex–post evaluation before an IA is launched. Future research could repeat our analysis for evaluations of non–regulatory activities or study the causal mechanisms behind our findings.

Type
Articles
Information
European Journal of Risk Regulation , Volume 7 , Issue 2 , June 2016 , pp. 388 - 403
Copyright
Copyright © Cambridge University Press 2016

Access options

Get access to the full version of this content by using one of the access options below. (Log in options will check for institutional or personal access. Content may require purchase if you do not have access.)

References

1 Communication to the Commission from Mrs. Schreyer in agreement with Mr. Kinnock and the President, “Focus on results: strengthening evaluation of Commission activities”, SEC(2000)1051; Communication to the Commission from Ms Grybauskaité in agreement with the President, “Responding to strategic needs: Reinforcing the use of evaluation”, SEC(2007)213; Commission Communication, “Smart Regulation in the European Union”, COM(2010)543 final; Commission Communication, “Strengthening the foundations of Smart Regulation – improving evaluation”, COM(2013)686 final.

2 Commission Staff Working Document, “Better Regulation Guidelines SWD(2015)111 final.

3 Ibid., p. 4. 4 From 2010 until 2014 the Better regulation agenda was called ‘Smart Regulation’. For the sake of consistency, in this article we only use the name Better Regulation, which was used in official communication before 2010 and is used again since 2015.

5 Radaelli, Claudio & Meuwese, Anne, “Hard questions, hard solutions: Proceduralisation through impact assessment in the EU”, 33 West European Politics (2010), pp. 136 et sqq., at pp. 137-40CrossRefGoogle Scholar; Commission SWD, “Better Regulation Guidelines”, supra note 2, at pp. 7-9.

6 Commission Communication, “Impact Assessments”, COM(2002)276 final, at p. 2.

7 Fitzpatrick, Teresa, “Evaluating Legislation: An alternative approach for evaluating EU Internal Market and Services law” 18 Evaluation (2012), pp. 477 et sqq., at p. 478.Google Scholar

8 Commission Communication, “improving evaluation”, supra note 1, at pp. 2-3; Commission Staff Working Document, “Better Regulation Toolbox”, complementing SWD(2015)111, at pp. 288.

9 Directorate-general for Internal Market and Services, “DG MARKT Guide to Evaluating Legislation”, (2008), at p. 51; Directorate–general Information Society and Media, “Evaluating Legislation and Non–Spending Interventions in the Area of Information Society and Media”, (2011) at p. 17.Google Scholar

10 Communication for the Commission from the President and Mrs Schreyer, “Evaluation standards and good practice”, COM(2002)2567 final, at p. 9.

11 The Evaluation Partnership, Evaluation of the Commission's Impact Assessment System, (Brussels: Secretariat–General of the European Commission 2007), at p. 86.Google Scholar

12 Impact Assessment Board, “Impact Assessment Board Report for 2013” available on the internet at http://ec.europa.eu/smart-regulation/impact/key_docs/docs/iab_report_2013_en.pdf (accessed 28-09-2015), at p. 7.

13 Smismans, Stijn, “Policy Evaluation in the EU: The Challenges of Linking Ex Ante and Ex Post Appraisal” 6 European Journal of Risk Regulation (2015), pp. 6 et sqq., at p. 19.Google Scholar

14 e.g. Cecot, Caroline, Hahn, Robert, Renda, Andrea and Schrefler, Lorna, “ An evaluation of the quality of impact assessment in the European Union with lessons for the US and the EU” 2 Regulation and Governance (2008), pp. 405 et sqq.;CrossRefGoogle Scholar Fransesco, Fabrizio de, Radaelli, Claudio M. and Troeger, Vera E., “Implementing regulatory innovations in Europe: the case of impact assessment” 19 Journal of European Public Policy (2012), pp. 491 et sqq.;Google Scholar Meuwese, Anne C.M., Impact Assessment in EU Lawmaking, (Zutphen: Wöhrmann Print Service 2008);Google Scholar Renda, Andrea, Impact Assessment in the EU: the state of the art and the art of the state, (Brussels: Centre for European Policy Studies 2006);Google Scholar Torriti, Jacopo and Löfstedt, Ragnar, “The first five years of the EU Impact Assessment system: a risk economics perspective on gaps between rationale and practice” 15 Journal of Risk Research (2012), pp. 169 et sqq. CrossRefGoogle Scholar

15 e.g. Fitzpatrick, , “Evaluating Legislation”, supra note 7; Elliot Stern, “Evaluation policy in the European Union and its institutions” New Directions for Evaluation (2009), pp. 67 et sqq.Google Scholar; Højlund, Steven, “Evaluation use in evaluation systems – the case of the European Commission”, 20 Evaluation (2014), pp. 428 et sqq. Google Scholar

16 Smismans, “Policy Evaluation in the EU”, supra note 13, at p. 19.

17 Smismans, “Policy Evaluation in the EU”, supra note 13, at p. 19

18 Ibid., at p. 22

19 Ibid., at p. 7

20 Fitzpatrick, “Evaluating Legislation”, supra note 7, at p. 478.

21 Communication to the Commission “Focus on Results”, supra note 1, at p. 6.

22 Luchetta, Giacomo, “Impact Assessment and the Policy Cycle in the EU”, 4 The European Journal of Risk Regulation 2012, pp. 562.Google Scholar

23 Communication for the Commission, “Evaluation standards and good practice”, supra note 10, at p. 7; European Commission, “Better Regulation Guidelines”, supra note 2, at p. 49.

24 European Commission, “Decision of the President of the European Commission on the establishment of an independent Regulatory Scrutiny Board” C(2015)3263 final.

25 Fitzpatrick, “Evaluating Legislation”, supra note 7, at p. 478.

26 Communication to the Commission, “Responding to Strategic needs”, supra note 1, at p. 10

27 Commission Communication, “Smart Regulation”, supra note 1, at p. 3; Commission Communication, “Improving evaluation”, supra note 1, at p. 4; Commission SWD, “Better Regulation Toolbox”, supra note 8, at p. 71.

28 High Level Group for Better Regulation. “Ex-post evaluation – final report”, at p. 12, available on the internet at http://ec.europa.eu/smart-regulation/impact/docs/wg3_report_ex_post_evaluation_en.pdf (accessed 28-09-2015).

29 Commission Communication, “Smart Regulation”, supra note 1, at p. 5.

30 Hartlapp, Miriam, Metz, Julia and Rauh, Christian, “Linking Agenda Setting to Coordination Structures: Bureaucratic Politics inside the European Commission”, 35 Journal of European Integration 2013, pp. 430.CrossRefGoogle Scholar

31 Meuwese, Anne and Gomtsyan, Suren, “Regulatory scrutiny of subsidiarity and proportionality” 22 Maastricht Journal of European and Comparative Law (2015), pp. 483 et ssq. at pp. 490-491.Google Scholar

32 Commission SWD, “Better Regulation Toolbox”, supra note 8, at p. 7; European Commission, “Regulatory Scrutiny Board”, supra note 23, at pp. 2-4.

33 Commission SWD, “Better Regulation Guidelines, supra note 2, at p. 30.

34 DG Markt, “Guide to Evaluating Legislation”, supra note 9, at p. 21.

35 Ibid., at p. 58.

36 DG INFSO, “Evaluating legislation”, supra note 9, at p. 17.

37 Ibid., at p. 12.

38 European Court of Auditors, Impact assessments in the EU institutions: do they support decision-making? (Luxembourg: European Court of Auditors 2010) Special report no. 3, at p. 40.Google Scholar

39 Cecot et. al., “Quality of impact assessments in the European Union”, supra note 14, at p. 409.

40 Gestel, Rob van and Vranken, Jan, ‘Assessing the accuracy of ex ante evaluation through feedback research: A case study”, in Verschuuren, J. (Ed.), The impact of legislation: A critical analysis of ex ante evaluation (Leiden, Boston: Martinus Nijhoff 2009), pp. 199 et sqq., at p. 199CrossRefGoogle Scholar.

41 Ibid., at pp. 225-228.

42 The Evaluation Partnership, Evaluation of the Commission's Impact Assessment System, supra note 11, at p. 87; Smismans, “Policy Evaluation in the EU”, supra note 13, at p. 13.

43 Bastiaan de Laat and Kevin Williams, ‘Evaluation use within the European Commission: lessons for the Commissioner’, in: Marlène Läubli Loud and John Mayne (eds.), Enhancing Evaluation Use: Insights from Internal Evaluation Units. (Sage: London 2014), pp. 147 et sqq., at p. 168.

44 Ibid., at p. 150.

45 e.g. De Laat and Williams, ‘Evaluation use within the European Commission’, supra note 40; Marlène Läubli Loud and John Mayne (eds.), Enhancing Evaluation Use: Insights from Internal Evaluation Units. (Sage: London 2014), at p. 3; Højlund, Steven, “Evaluation use in the organizational context – changing focus to improve theory”, 20 Evaluation (2014), pp. 26 et sqq.;Google Scholar Højlund, Steven, “Evaluation use in evaluation systems – the case of the European Commission”, 20 Evaluation (2014), pp. 428 et sqq.;CrossRefGoogle Scholar Damien Contandriopoulos and Astrid Brousselle, “Evaluation models and evaluation use”, 18 Evaluation 2012, pp. 61 et sqq.;Google Scholar Kim Forss, K., Rebien, Claus C., and Carlsson, Jerker, “Process use of evaluation: Types of use that Precede Lessons Learned and Feedback”, 8 Evaluation 2002, pp. 29 et sqq. Google Scholar

46 Loud and Mayne, Enhancing Evaluation Use, supra note 42, at p. 7.

47 Smismans, “Policy Evaluation in the EU”, supra note 13, at pp. 15-22; De Laat and Williams, ‘Evaluation use within the European Commission’, supra note 40, at pp. 158-62.

48 De Laat and Williams, ‘Evaluation use within the European Commission’, supra note 40, at pp. 158-160; Højlund 2014, “Evaluation use in evaluation systems”, supra note 42, at p. 429.

49 Smismans, “Policy Evaluation in the EU”, supra note 13, at p. 19.

50 Emanuela Bozzini and Jo Hunt, “Bringing Evaluation into the Policy Cycle CAP Cross Compliance and the Defining and Redefining of Objectives and Indicators”, 6 European Journal of Risk Regulation 2015, pp. 57 et sqq., at pp. 64-65;Google Scholar Mergaert, Lut and Minto, Rachel, “Ex Ante and Ex Post Evaluations: Two Sides of the Same Coin? The Case of Gender Mainstreaming in EU Research Policy”, 6 European Journal of Risk Regulation 2015, pp. 47 et sqq., at p. 53.Google Scholar

51 De Laat and Williams, ‘Evaluation use within the European Commission’, supra note 40, at p. 162.

52 Ibid., at p. 162.

53 Commission SWD, “Better Regulation Guidelines, supra note 2, at p. 48.

54 Smismans, “Policy Evaluation in the EU”, supra note 13, at p. 18; Luchetta, “Impact Assessment and the Policy Cycle in the EU”, supra nota 22, at p. 571.

55 Ibid., at pp. 17-23.

56 Ibid., at p. 18.

57 Ibid., at p. 23.

58 Supra note 52.

59 Balthasar, Andreas, “Institutional Design and Utilization of Evaluation A Contribution to a Theory of Evaluation Influence Based on Swiss Experience”, 33 Evaluation review 2009, pp. 226 et sqq., at pp. 231-236.CrossRefGoogle ScholarPubMed

60 De Laat and Williams, ‘Evaluation use within the European Commission’, supra note 40, at p. 167; European Policy Evaluation Consortium, Study on the use of evaluation results in the European Commission, (Brussels: EPEC 2005), at p. 61.Google Scholar

61 To implement this principle, we excluded all legislation initiated by the following DGs and services: DEVCO, ECHO, FPI, ENLARG.

62 With this we refer to any regulation or directive which is only binding for EU civil servants or for the legal behavior of the institutions of the EU.

63 Smismans, “Policy Evaluation in the EU”, supra note 13, at p. 13.

64 http://ec.europa.eu/smart-regulation/evaluation/search/search.do, (accessed 28-09-2015).

65 http://Bookshop.europa.eu, (accessed 28-9-2015).

66 Commission Staff Working Document, “the evaluation of the Union's finances based on the results achieved”, SWD(2012) 383 and Commission Staff Working Document, “the evaluation of the Union's finances based on the results achieved”, COM(2013)228.

67 Mastenbroek, Ellen, Voorst, Stijn van and Meuwese, Anne, “Naar een regelgevingcyclus? Evaluatie in de Europese Unie” 29 Regel-Maat 2014, pp. 212 et sqq., at pp. 221-222.Google Scholar

68 http://ec.europa.eu/smart-regulation/impact/ia_carried_out/cia_2014_en.htm, (accessed 28-09-2015).

69 Such as IAs for Communications, decisions, action plans and recommendations.

70 For more details, see footnote 61 and 62 above.

71 Impact Assessment Board, “Report for 2013”, supra note 12, at p. 7.

72 The legislative observatory can be found at http://www.europarl.europa.eu/oeil/search/search.do? (accessed 28-09-2015).

73 The research was done with Adobe Acrobat Reade, using an advanced search on the folders containing the IAs. Folders were divided per year.

74 Using the three keywords ‘report, ‘review’ and ‘study’ could generate 10.000+ hits per year. This could lead to roughly 100+ hits per IA.

75 European Commission, “Impact Assessment Guidelines”, SEC(2009)92 final, at p. 8.

76 Commission SWD, “Better Regulation Toolbox”, supra note 8, at p. 29.

77 Widmer, T., ‘Instruments and procedures for assessing evaluation quality: a Swiss perspective’, in: R., Schwartz & J., Mayne (Eds.), Quality Matters: Seeking Confidence in Evaluating, Auditing and Performance Reporting (New Brunswick: Transaction 2005), pp. 41 et sqq., at p. 43.Google Scholar

78 Cecot et. al., “Quality of impact assessments in the European Union”, supra note 14, at p. 418

79 Commission SWD, “Better Regulation Toolbox”, supra note 8, at pp. 49-65.

80 Mastenbroek et. al., “Evaluatie in de Europese Unie”, supra note 63, at pp. 223-225.

81 Mastenbroek et. al., “Evaluatie in de Europese Unie”, supra note 63, at pp. 223-225.

82 Since we have five explanatory conditions, we would need fifty positive cases for regression analysis.

83 Charles Ragin, Redesigning social inquiry: fuzzy sets and beyond, (Chicago: University Press 2008), at p. 9.

84 Ragin, Fuzzy sets and beyond, supra note 77, at p. 85.

85 Kogut, Bruce, McDuffie, John Paul, and Ragin, Charles, “Prototypes and strategy: assigning causal credit using fuzzy sets”. 1 European Management Review (2004), pp. 114 et sqq., at p. 123.Google Scholar

86 Ragin, Fuzzy sets and beyond, supra note 77, at. p. 36.

87 The Evaluation Partnership, Evaluation of the Commission's Impact Assessment System, supra note 11, at p. 12.

88 Furthermore, note that some IAs refer to ex-post evaluations of individual policy programmes or action plans. Such evaluations were not counted even if the instrument they study has a legal basis.

89 Commission SWD, “Better Regulation Guidelines”, supra note 2, at p. 49.

90 Ragin, Fuzzy sets and beyond, supra note 77, at. p. 102.

91 Ragin, Fuzzy sets and beyond, supra note 77, at. p. 120. 92 Smismans, “Policy Evaluation in the EU”, supra note 13, at pp. 17-22.

93 Ragin, Fuzzy sets and beyond, supra note 77, at. p. 125.

94 Bozzini and Hunt, “Bringing Evaluation into the Policy Cycle”, supra note 47, at pp. 64-65.

95 Mergaert and Minto, “Ex Ante and Ex Post Evaluations” supra note 47, at p. 53. 96 European Parliament resolution of 8 June 2011 on guaranteeing independent impact assessments (2010/2016(INI)), available on internet at http://www.europarl.europa.eu/sides/getDoc.do?pub-Ref=-//EP//NONSGML+TA+P7-TA-2011-0259+0+DOC+PDF+V0//EN, (accessed 28-09-2015).

97 Inter–institutional Agreement on better Law–making, OJ 2003 C 321/01.

98 European Commission, “Proposal for an Interinstitutional Agreement on Better Regulation” COM(2015)216 final, at p. 6.

99 Supra note 28; Commission SWD, “Better Regulation Toolbox”, supra note 8, at p. 17. It should however be noted that the Commission leaves some discretionary room to ignore the ‘evaluate first’ principle if it is ‘justified by political demands on the Commission’ according to p. 256 of the Toolbox.

100 Supra note 75.

101 Bovens, Mark, 't Hart, Paul and Kuipers, Sanneke, ‘The politics of Policy Evaluation’, in: Goodin, R. E., Rein, M. & Moran, M. (Eds.), The Oxford Handbook of Public Policy, (Oxford: Oxford University Press 2008), pp. 320 et ssq., at. p. 320.Google Scholar

102 Supra note 8, at pp. 28 and 254.