CM(2008)156 27 October 20081
1042 Meeting, 25-27 November 2008
1 General questions
1.6 Evaluation Policy for the Council of Europe
Examined by the GT-REF.INST at its meeting of 13 November 2008
1. Introduction 2
1.1 Definition of the evaluation concept 2
1.2 Objectives of evaluation 3
2. Evaluation benchmarking: Policy and practice in other international organisations 3
3. Evaluation in the Council of Europe 4
3.1 Evaluation methods 4
4. Roles and responsibilities 4
4.1 The Committee of Ministers 4
4.2 The Steering Committees and subordinate bodies 5
4.3 The Secretary General 5
4.4 Operational Management 5
4.5 Programme Coordinators and Projects Managers 5
4.6 The Evaluation Entity 6
5. Evaluation management 6
5.1 Quality management and evaluation capacity 7
5.2 Evaluation budget 7
5.3 Follow-up of evaluation 8
5.4 Council of Europe transparency and disclosure policy 8
1. This document presents a proposal for a general framework for a comprehensive Council of Europe evaluation policy. The purpose of the policy is to establish a common institutional basis for the Council of Europe evaluation function, standardise and professionalise assessment methods and facilitate dialogue with relevant partners and stakeholders.
This document is based on the assumption that evaluation should be an integral part of attempts towards improved efficiency and innovation.
2. The policy seeks to increase transparency, coherence and efficiency in generating and using evaluative knowledge for organisational learning and effective results-oriented management and to improve accountability, in conformity with Chapter V of the Warsaw Action Plan.
3. The policy needs to be consistent with internationally accepted evaluation norms, standards and good practices and will be built upon a general approach towards evaluation of the joint Council of Europe/European Commission programmes.
4. The Council of Europe will build on its current capacity and commitment to applying international good practice. Improvement in the management practices and the functioning of the Organisation is expected to result from the application of the proposed policy in the medium term. Envisaged improvements are:
· more systematic use of evaluation;
· regular reporting to the Committee of Ministers and the Council of Europe senior management on evaluation activity and its effect;
· follow-up of evaluation findings and recommendation, including their use in the results-based planning, programming and budgeting process;
· improving institutional learning and knowledge-sharing;
· harmonisation of evaluation practices and methods within the Council of Europe, regardless of funding source;
· improved evaluation capacity and skills in evaluation and self-evaluation methodologies.
5. The policy draws from and is aligned with criteria for launching, discontinuing and evaluating projects approved by the Committee of Ministers in January 2007 (CM(2006)101 final).
6. The scope and methods of evaluation do have significant financial implications. In view of scarcity of resources, a step-by-step process is therefore proposed, starting with a more systematic and coherent self-evaluation, and envisaging a progressive shift towards independent evaluation. A roadmap with targets for the next few years should be developed.
1.1 Definition of the evaluation concept
7. Evaluation is an assessment, as systematic and impartial as possible, of a project, programme or entire strand of activities under a single thematic or institutional heading. An evaluation should provide evidence-based information that is credible, reliable and useful, thereby permitting the timely incorporation of findings, recommendations and lessons into the decision making process at the corporate, programme and project levels.
8. It aims at determining the relevance, added-value, effectiveness, efficiency, impact and sustainability of the Council of Europe interventions, based on agreed criteria (CM(2006)101 final) and benchmarks among key partners and stakeholders.
9. Evaluation focuses on the analysis of expected and achieved accomplishments, examining the results chain, processes, contextual factors and causality, in order to ascertain the degree of achievement or the lack thereof.
10. Evaluation differs from other forms of assessment, in particular from:
· monitoring (management’s continuous examination of progress achieved during the implementation of a project or a programme in order to track compliance with the plan and to take necessary decisions to improve performance);
· audit (an independent assessment of the adequacy of management controls to ensure the economical and efficient use of resources and improve the organisation’s operations, in particular the effectiveness of risk management, as well as the adequacy of systems, processes and performance. The tasks of the Internal Auditor of the Council of Europe are defined in Instruction No. 52 of 1 October 2005.).
1.2 Objectives of evaluation
11. Evaluation serves four main purposes: it improves accountability, supports management and decision-making, facilitates dialogue with partners and stakeholders, drives learning and innovation.
12. Evaluation enhances accountability by reporting on Council of Europe activities to member states, partners and donors that have (co-) financed the activity and to other stakeholders of the Council of Europe.
13. Evaluation feeds into management and decision-making processes regarding the development of policies and strategies. It informs the planning, programming, budgeting, implementation and reporting cycle. Evaluation provides recommendations to programme coordinators, project managers and management at all levels, as well as to Council of Europe partners and donors.
14. Recommendations aim at improving the institutional relevance and the achievement of results, providing stakeholders satisfaction, optimising the use of resources, and maximising the impact of the contribution provided. It facilitates future planning of activities with other partners and their political assessment by stakeholders.
15. Evaluation drives organisational learning and innovation. This involves the creation of an environment that engages staff in creative ways to learn how to improve the Council of Europe work. In this context, evaluation is an instrument for making the Council of Europe policies, programmes, projects and organisational entities more effective through the provision of useful feedback and a commitment to act on that feedback.
2. Evaluation benchmarking: Policy and practice in other international organisations
16. The comparison with other organisations has provided some useful elements to be considered for a Council of Europe evaluation policy: Proposals in this document are notably derived from structures, policies and practices observed in the European Commission, the OECD, UNESCO, UNIDO and the CEB.
- Structure/responsibilities: Frequently, an evaluation group/unit/department has been created at central level to supervise and/or execute an independent evaluation function.
- Scope and timing: Evaluations mostly include independent mid-term, final and ex-post evaluations. In addition, self-evaluation is practiced frequently with differing modalities. Evaluations can be country-specific/regional, thematic, related to projects and programmes
- Evaluation standards and criteria: Commonly, the DAC (Development Assistance Committee) (OECD) criteria are applied, often complemented by specific organisational guidelines. Most organisations follow closely recommendations of the vast network of international evaluation associations concerning principles, practice and ethics of evaluation and regularly participate in conferences among evaluators and evaluation departments.
- Transparency/public documents: Most organisations publish their evaluation policy guidelines or strategy documents as well as selected evaluations and/or a periodical report about evaluation.
3. Evaluation in the Council of Europe
17. The evaluation policy has to provide an operational framework that serves different needs and is aimed at different levels of the organisation.
18. The main types of evaluation that could be carried out within the Council of Europe are the following:
(a) Organisational evaluation (functioning of institutional arrangements, field offices, Council of Europe Information Offices and Partial Agreements);
(b) Thematic and cross-cutting evaluation;
(c) Regional or Country programme evaluation;
(d) Evaluation of projects and programmes of the Programme of Activities (PoA).
19. Organisational evaluation is rather large-scale and exceptional and will therefore be discarded in the framework of this document. The other types of evaluation are, however, needed to adapt project management and country, regional or thematic strategies to a changing environment. Project and programme evaluation is part of the project management cycle. In the framework of the PoA, evaluation is essential to project/programme identification and formulation. It feeds directly into project/programme assessment and is closely connected to monitoring/reporting. Evaluation quality depends on the other stages of the programme/project cycle: realism and logic of objectives as well as definition of meaningful indicators; quality of information produced under monitoring/reporting and congruence between rigorous planning and flexibility during implementation.
3.1 Evaluation methods
20. Project and programme evaluation could take two forms: self-evaluation and independent evaluation.
21. Self-evaluation involves periodic progress reviews of projects or programmes carried out by those responsible for implementation. Self-evaluation builds upon monitoring and reporting and takes place according to the “Evaluation Guidelines” to be established upon the adoption of the evaluation policy. Self-evaluation is the vehicle for steering corrective action by line management.
b. Independent evaluation
22. Independent evaluation provides an independent view on a given entity under evaluation, such a project, a programme or an entire strand of activities under a thematic or institutional heading. The ‘establishing criteria for projects’ CM(2006)101 final2 will be used as a basis for selecting projects that require independent evaluation.
23. Independent evaluation might be internally led by members of the Evaluation Entity or externally led by independent consultants, depending on the capacity and competence required.
4. Roles and responsibilities
24. The roles and responsibilities concerning the key constituents of the Council of Europe evaluation system are identified below.
4.1 The Committee of Ministers
25. The Committee of Ministers:
· approves the evaluation policy;
· draws on the findings and recommendations of evaluation for oversight and approval of corporate policy, strategy and the Programme of Activities.
4.2 Steering Committees and subordinated bodies
26. The Steering Committees and subordinate bodies will, in accordance with Resolution Res(2005)47:
· review the summary of results of evaluation as submitted by the Secretariat;
· draw on evaluation findings to propose improvements of the quality of programmes, guide strategic decision-making on future programming and positioning, and communicate their position and/or recommendations to the Committee of Ministers.
4.3 The Secretary General
27. The Secretary General is accountable for the Council of Europe results, and:
· ensures compliance with the evaluation policy as integral to effective accountability across the organisation;
· ensures the integrity of the evaluation function and its independence from operational management;
· reports annually to the Committee of Ministers on the function, findings and recommendations of evaluations, on compliance, quality assurance, and follow-up to evaluation conducted;
· provides sufficient resources and capacity for evaluation in the Organisation;
· ensures that the Council of Europe prepares a management response to evaluation;
· ensures that operational management responds to and uses evaluation in its operational, strategic and oversight functions and that appropriate follow-up to the findings and recommendations of evaluation is taken by the relevant DGs.
4.4 Operational Management
28. Operational management:
· provides sufficient resources and capacity for evaluation in its areas of responsibility;
· ensures the effective monitoring of implementation and performance of programmes to generate relevant, timely information for management for results and evaluation;
· identifies, with key partners and stakeholders, priority areas for evaluation when preparing the programme, and designing and implementing a strategic Evaluation Plan in its areas of responsibility;
· supports evaluation by ensuring that all necessary information is provided, and that programme coordinators and project managers cooperate fully in evaluation;
· ensures implementation of the evaluation recommendations in the areas of its responsibility.
4.5 Programme Coordinators and Project Managers
29. Programme coordinators will, within the framework of the programmes under their responsibility, be required to:
· actively contribute to preparation of the terms of references, including the determination of the most suitable approach;
· contribute to the preparation, execution and follow-up of evaluation and provide the evaluators with a complete information dossier well in advance of the evaluation;
· in the case of Joint European Commission/Council of Europe programmes, the programme coordinator is the evaluator’s central access point. In his/her coordinating role and responsibility for programme monitoring, he/she assumes responsibility for the preparation of the information dossier;
· ensure that the necessary evaluation funds are properly allocated in the project budget during the formulation phase;
· contribute to and/or coordinate the preparation of management responses to evaluation;
· assume primary responsibility for the follow-up to evaluation recommendations.
30. Project managers will conduct self-evaluation of projects under their responsibility. They will ensure that the projects they manage are subject to regular evaluation in conformity with “Evaluation Guidelines” to be developed upon the adoption of the present policy.
4.6 The Evaluation Entity
31. The Evaluation Entity is the custodian of the evaluation function. The Evaluation Entity should:
(a) in respect of governance and accountability
· prepare and periodically review and update the Council of Europe policy for evaluation;
· draft and submit its (multi-) annual evaluation plan to the Secretary General;
· report annually on the function, findings and recommendations of evaluations, on compliance, quality assurance, and follow-up to evaluation conducted;
· maintain a record of management responses to all evaluations;
· alert senior management to emerging evaluation-related issues of corporate significance.
(b) conduct evaluation
· develop an agenda, contingent upon available resources, for independent evaluation, based on consultations with the Committee of Ministers, senior management and other stakeholders, and in response to emerging issues that the Evaluation Entity may identify;
· conduct evaluation in conformity with the Annual Evaluation Plan;
· ensure that evaluation provides targeted and representative coverage of the Council of Europe activities and results, and that mandatory evaluation is carried out.
(c) in respect of quality assurance
· set evaluation standards for planning, conducting and using evaluation, developing and disseminating methodology and establishing the institutional mechanisms for applying the standards;
· support the setting of frameworks and standards for monitoring in the context of the Council of Europe result-based management system to facilitate the evaluation of programmes, projects and activities;
· provide technical support to decentralised evaluation and support quality assurance of the evaluations conducted by external groups.
(d) in respect of knowledge management
· maintain a publicly accessible repository of evaluation reports;
· distil evaluation findings and lessons for dissemination in appropriate formats for targeted audiences;
· support the development of learning groups of practice in evaluation by establishing and supporting knowledge networks.
(e) in respect of Council of Europe reform
· support the harmonisation of the evaluation function throughout different parts of the Organisation;
· use the experience of the Council of Europe and other international organisations to advance the science, practice, quality and usefulness of evaluation.
5. Evaluation management
32. The annual work programme makes sure that evaluations are chosen and undertaken in a transparent and timely manner so that they provide decision-making with relevant and timely information.
33. The annual work programme describes the planned activities of the Evaluation Entity over a given year (N+1). The Evaluation Entity drafts this work programme in the framework of budgetary discussions for the following year in consultation with the Council of Europe programme coordinators, project managers, thereafter submitting it for approval to the Secretary General. The Evaluation Entity updates the annual work programme in order to accommodate any necessary adjustments.
5.1 Quality management and evaluation capacity
34. Implementation of the evaluation policy will require an internal quality assurance system and rigour in the conduct of evaluation.
35. The Evaluation Entity will develop a set of “Evaluation guidelines” to guide the practice of evaluation. All staff involved in evaluation will have access to these guidelines and be able to use them for their evaluation work. All evaluation should meet minimum standards. The Evaluation Entity will review evaluation reports on a regular basis to ensure standards are met and will report on progress towards improving the quality of evaluation on an annual basis.
36. The Evaluation Entity will be the focal point of evaluation know-how in the Council of Europe. In collaboration with the Council of Europe Human Resources Directorate, it will design and carry out training for the Council of Europe staff on subjects that are relevant to evaluation.
5.2 Evaluation budget
37. The current evaluation unit within the Directorate of Strategic Planning, which carries out both self-evaluation and oversees external evaluation, is made up of ½ A2/3 and ¼ A4. This capacity is insufficient to carry-out a meaningful and sustainable Council of Europe-wide evaluation policy. The Evaluation Entity would therefore need to be provided with sufficient human and financial resources as well as the necessary competences to perform its tasks or the scope of its tasks would need to be adapted in the light of the resources available.
38. The annual evaluation work programme would be included in the Evaluation Entity PoA PMM logframe with the corresponding budget forecast.
39. Three components of the Council of Europe evaluation budget would be managed by the Evaluation Entity, namely:
· appropriations to finance the proposed in-house, independent evaluation and/or contracted external evaluation;
· appropriations for specific, targeted, external evaluation, to be included in the draft PoA/Budget and the draft annual evaluation plan;
· resources already included in third party financing arrangements (e.g. Council of Europe/European Commission Joint Programmes).
40. Appropriations foreseen for routine self-evaluation in the Council of Europe PoA (essentially staff cost only) would be managed by the appropriate administrative entities.
41. Funding for evaluation will have to be provided from all available sources: Ordinary Budget, Joint Programmes and Voluntary Contributions. Currently, funds for evaluation are only provided for in Joint Programmes. An average of 2-3% of total project funds is made available for evaluation. Average real costs for an ex-post or final evaluation of a programme are approximately €30 000-€50 000. As far as Ordinary Budget projects/programmes are concerned, there are currently no budget provisions for evaluation. The only available resources are currently some consultancy provisions which would cover maximum 2 evaluations per year in total. Here, the capacity for independent evaluation should be progressively increased in the coming years. The following financing method is proposed:
42. Having in mind the number of programmes and projects in the PoA and available resources, a targeted selection of programmes/projects has to be made which are to be evaluated. Taking into account the situation summarised above, a minimum threshold of a total project cost of €1 million seems appropriate (to be revised later). For any ongoing or new project/programme of this size, therefore, a minimum amount of €30 000-€50 000 should be foreseen for evaluation. A special account (“Evaluation Provision”) should be created allowing, in addition to the ex-post evaluation of large projects, for targeted ad hoc evaluations – even during project implementation – in order to address specific aspects or problems.
This special account could be financed by Ordinary Budget provisions and/or by voluntary contributions. Funding of certain thematic or country-specific evaluations could possibly also be considered by means of the newly established Human Rights Trust Fund.
5.3 Follow-up of evaluation
43. Proper and efficient evaluation implies that there is a clear intent to use the evaluation findings, recommendations and lessons learned. The Evaluation Policy recognise the need for having a system in place that enables it to respond to, and follow-up on, the recommendations of an evaluation.
44. Evaluations will need a management response. The Evaluation Entity will maintain a system to track management responses to evaluations. The follow-up to self-evaluations will come under responsibility of programme coordinators and project managers.
45. Follow-up to independent evaluation will come under the responsibility of the Evaluation Entity. The Evaluation Entity will transmit the evaluation report the Secretary General together with a management response sheet. This sheet will provide for tracking, for each recommendation, comments of acceptance or non-acceptance of evaluation recommendations, deadlines and action taken by those responsible for
46. The Evaluation Entity will make the lessons learned from evaluation available throughout the organisation as well as to field offices. To facilitate wider use and dissemination of evaluation findings, the executive summary of all strategic evaluation will be translated into the two official languages of the Council of Europe, and if appropriate, into a national language.
5.4 Council of Europe transparency and disclosure policy
47. The transparency of the evaluation process is an important aspect of ensuring that evaluation exercises are extensively used by staff and at the governance level. The Secretary General is responsible, when appropriate, for making complete evaluation reports accessible to key partners and stakeholders.
48. This evaluation policy, the annual evaluation plan, the annual evaluation report, terms of references for independent evaluations, will be posted in the relevant pages of the Council of Europe website and major evaluation contracts will be awarded through a process of competitive tendering.
Note 1 This document has been classified restricted until examination by the Committee of Ministers.
Note 2 Approved at the Deputies' 984th meeting (17 and 18 January 2007, item 1.9).