Standards

Why RTDI evaluation? Why RTDI evaluation standards?

 Research and technology policy comprises all public initiatives which involve the organization of research and innovation systems — i.e. not only public programmes, policies, strategies and regulations but also the "landscape" of institutions which carry out research and develop technology. What is special about research and technology policy in comparison with other policy areas is that programmes, policies and institutions legitimize their existence on the basis of certain market and system failure phenomena inherent in research, technological development and innovative dynamics. Research and technology policy should be in a position to show that investments in this field are worthwhile, particularly within a European context. RTDI decision-makers require instruments and techniques appropriate to the growing need for information, for example about the functioning of RTDI systems, the quality of research and technology vis-a-vis international standards, and the effectiveness of research and technology funding.

Since it is essential to prove that RTDI investments are worthwhile, it is necessary to provide a basis for continuing assessment in RTDI policies. "Evaluation", an umbrella term covering a range of different techniques, methods and measures, has become internationally established as a tool for this purpose. As such, it provides information for politicians, programme managers and the interested general public on the suitability of specific initiatives for achieving objectives in the public interest and for overcoming market and system failures. Consequently, a developed culture of evaluation is an integral part of strategically oriented research and technology policy that is continuing to grow. A good culture of evaluation is both a pre-requisite for and a consequence of good policy: in other words, the process must be efficient, transparent and fair.

However, in order to develop the culture of RTDI evaluation and root it deeply into an RTDI system and RTDI culture as a whole, RTDI evaluation should be, first and foremost, standardized. Only then can it be used continuously and further upgraded and developed. Having a regional approach to standardization only makes it more efficient and effective, and raises the quality of the standards themselves.

Apart from providing a tool for RTDI assessment, a standardized approach to evaluation provides a starting point for developing and strengthening the capacities of different stakeholders and target groups, ranging from policy makers to RTDI performers, which enables them to adapt their activities to meet ever-changing market requirements and prove that the resulting outcomes are worth the investments.

 Let’s go back to the evaluation itself. For better illustration, here is a reflection on, first of all, the functions of RTDI evaluation. Namely, different types of evaluation have different functions and effects. What they all have in common, however, is the fact that they can fulfill, or help to fulfill, a number of tasks within the policy cycle. Evaluation can, depending on the basic conditions provided, fulfill various functions:

• Legitimising: e.g. justifying the use of public funds;
• Informing: providing the public with information on how public funds are being used and to what effect;
• Information-oriented learning: for those funding and/or implementing programmes, decision-makers in the field of technology policy, scientists, etc.;
• Steering: for the establishment of policy objectives, planning measures, etc., for the future;
• Controlling: as in private enterprise.

 Evaluation can also fulfill a mediating function between the competing interests of various players in research and technology policy. To avoid misunderstandings between the evaluators and the institutions commissioning an evaluation, it is necessary for them to discuss the function of the evaluation and make this information transparent right from the beginning.

 Here is a list of the benefits that RTDI standards of evaluation bring to different groups, including evaluators and institutions commissioning evaluations, as well as those to be evaluated. 

First of all, they provide a framework and a set of guidelines for the evaluation process, in turn supporting:

• Policy-makers, in designing programmes, formulating terms of reference (TORs), selecting evaluators, implementing evaluation results, and public relations work;
• RTD (programme-) management, in the setting up of monitoring systems, and assessment of individual projects;
• Those to be evaluated, in formulating project proposals, and planning their projects in terms of content and timing;
• Evaluators, in designing evaluation projects, positioning themselves vis-à-vis commissioning institutions, positioning themselves vis-à-vis those to be evaluated, and establishing a data base for evaluations.

In the context of the EVAL-INNO project, we are convinced that the standards we will produce will help us, first of all, create a pool of experts that will be apt to contribute to RTDI systems in their own countries and the SEE region. Our wider wish is that these experts will also be willing to work in the broader EU environment and that they will, in combination with all other project activities, aid in the further development of RTDI systems within individual countries.