Guidelines for Programme Evaluation and Benchmarking Manual

Guidelines for Programme Evaluation

A programme can be defined as a set of interdependent policy interventions, which covers sub-programmes, projects or measures aimed at achieving a specific, pre-defined objective. A programme usually has a fixed time period, its own financial administration, and a clear hierarchical structure. If a programme has RTDI specific aims, it can be subject to RTDI evaluations, for which the EVAL-INNO project team aims to develop guidelines.

Ideally, for RTDI programmes, an evaluation system would cover the following (adopted from FTeval’s Evaluation Standards in Research and Technology Policy):

• The times and objectives of different evaluation steps;
• The minimum methodological requirements;
• The consequences of the respective evaluations: How are the results to be translated into action?
• How evaluation would be carried out (external experts, and/or programme management or funding agency);
• How results are to be published;
• The transparency of decision-making criteria in project evaluation;
• Involvement of decision-makers;
• The guarantee of information collection from the beginning of the project/programme onwards;
• The guarantee of interaction between controlling, monitoring and evaluating.

Because of their underdeveloped evaluation cultures, such systems are virtually non-existent in many countries of the SEE region, which implies that RTDI programme evaluation guidelines should be transposed and adopted for the region, placing even more emphasis on the systemic improvement of conditions for innovation.

Within this activity, the EVAL-INNO project will prepare general guidelines and checklists for conducting sound RTDI programme evaluations as follows:

• Defining the contexts in which RTDI programme evaluation can be useful;
• Describing steps, techniques, measures, instruments etc. for programme evaluations, with a focus on practical issues;
• Organisation of a workshop to discuss draft guidelines;
• Finalisation of guidelines.

 It is planned that the guidelines will be made available in 2012.

 

Manual for Benchmarking

In EVAL-INNO, benchmarking innovation “intermediary” (or “bridging”) and “R&D based innovation organisations” are planned as a pilot exercise, based on a consensual Manual for Benchmarking, which is also intended for future use.

As stated in the EVAL-INNO project plan, the manual will be based on the methodological design applied in the RECORD network, which was based on best practices in the so-called centres of excellence research and technology organisations (RTOs). In a similar vein, the methodological steps of developing the EVAL-INNO Manual include:

• Definition of “intermediary” (or “bridging”) and “R&D based innovation organisations”, their general missions, functions and, if possible, taxonomies and tipologies;
• Discussion of knowledge generation, knowledge utilisation and knowledge diffusion functions in the context of the above organisations, and reaching consensus as to which related functions shall be defined as key function(s) — if possible, also linked with taxonomies/tipologies;
• Definition of internal, negotiated and external factors and factor groups that influence performance of the above key function(s);
• Developing benchmarks, i.e. quantitative performance metrics and qualitative practice measures, which can be used as a generic benchmarking frame;
• Developing questionnaires and interview templates for benchmarking exercises.

It is important to emphasise that good benchmarking means focused benchmarking, so reaching consensus about the above design among the EVAL-INNO team is important; this goal  is assisted by workshops dedicated to the development of the manual.

The manual, which shall cover a broad range of topics (see above) is planned to be made available in 2012, and will be first used in a pilot benchmarking exercise. At a later stage of the project, the manual and the results of the pilot benchmarking exercise will be widely disseminated.