definition of evaluation by different authors

If knowledge exchange events could be captured, for example, electronically as they occur or automatically if flagged from an electronic calendar or a diary, then far more of these events could be recorded with relative ease. Attempting to evaluate impact to justify expenditure, showcase our work, and inform future funding decisions will only prove to be a valuable use of time and resources if we can take measures to ensure that assessment attempts will not ultimately have a negative influence on the impact of our research. Assessment is the collection of relevant information that may be relied on for making decisions., 3. What are the methodologies and frameworks that have been employed globally to assess research impact and how do these compare? These . The Payback Framework enables health and medical research and impact to be linked and the process by which impact occurs to be traced. It is very important to make sure people who have contributed to a paper, are given credit as authors. 2005; Wooding et al. 60 0 obj << /Linearized 1 /O 63 /H [ 1325 558 ] /L 397637 /E 348326 /N 12 /T 396319 >> endobj xref 60 37 0000000016 00000 n Impact is derived not only from targeted research but from serendipitous findings, good fortune, and complex networks interacting and translating knowledge and research. Concerns over how to attribute impacts have been raised many times (The Allen Consulting Group 2005; Duryea et al. In endeavouring to assess or evaluate impact, a number of difficulties emerge and these may be specific to certain types of impact. Hb```f``e`c`Tgf@ aV(G Ldw0p)}c4Amff0`U.q$*6mS,T",?*+DutQZ&vO T4]2rBWrL.7bs/lcx&-SbiDEQ&. What are the reasons behind trying to understand and evaluate research impact? In 200910, the REF team conducted a pilot study for the REF involving 29 institutions, submitting case studies to one of five units of assessment (in clinical medicine, physics, earth systems and environmental sciences, social work and social policy, and English language and literature) (REF2014 2010). Wooding et al. CERIF (Common European Research Information Format) was developed for this purpose, first released in 1991; a number of projects and systems across Europe such as the ERC Research Information System (Mugabushaka and Papazoglou 2012) are being developed as CERIF-compatible. 0000001087 00000 n A taxonomy of impact categories was then produced onto which impact could be mapped. One way in which change of opinion and user perceptions can be evidenced is by gathering of stakeholder and user testimonies or undertaking surveys. 0000004731 00000 n To achieve compatible systems, a shared language is required. (2007), Nason et al. The definition problem in evaluation has been around for decades (as early as Carter, 1971), and multiple definitions of evaluation have been offered throughout the years (see Table 1 for some examples). Reviewing the research literature means finding, reading, and summarizing the published research relevant to your question. The case study of the Research Information System of the European Research Council, E-Infrastructures for Research and Innovation: Linking Information Systems to Improve Scientific Knowledge, Proceedings of the 11th International Conference on Current Research Information Systems, (June 69, 2012), pp. This distinction is not so clear in impact assessments outside of the UK, where academic outputs and socio-economic impacts are often viewed as one, to give an overall assessment of value and change created through research. 0000008591 00000 n n.d.). The criteria for assessment were also supported by a model developed by Brunel for measurement of impact that used similar measures defined as depth and spread. For more extensive reviews of the Payback Framework, see Davies et al. Classroom Assessment -- (sometime referred to as Course-based Assessment) - is a process of gathering data on student learning during the educational experience, designed to help the instructor determine which concepts or skills the students are not learning well, so that steps may be taken to improve the students' learning while the course is 2008; CAHS 2009; Spaapen et al. (2006) on the impact arising from health research. 0000346296 00000 n The Value of Public Sector R&D, Assessing impacts of higher education systems, National Co-ordinating Centre for Public Engagement, Through a Glass, Darkly: Measuring the Social Value of Universities, Describing the Impact of Health Research: A Research Impact Framework, LSE Public Policy Group. Not only are differences in segmentation algorithm, boundary definition, and tissue contrast a likely cause of the poor correlation , but also the two different software packages used in this study are not comparable from a technical point of view. Also called evaluative writing, evaluative essay or report, and critical evaluation essay . 2006; Nason et al. Where narratives are used in conjunction with metrics, a complete picture of impact can be developed, again from a particular perspective but with the evidence available to corroborate the claims made. What are the challenges associated with understanding and evaluating research impact? As part of this review, we aim to explore the following questions: What are the reasons behind trying to understand and evaluate research impact? RAND Europe, Capturing Research Impacts. Attempts have been made to categorize impact evidence and data, for example, the aim of the MICE Project was to develop a set of impact indicators to enable impact to be fed into a based system. If impact is short-lived and has come and gone within an assessment period, how will it be viewed and considered? In development of the RQF, The Allen Consulting Group (2005) highlighted that defining a time lag between research and impact was difficult. 10312. Evaluative research has many benefits, including identifying whether a product works as intended, and uncovering areas for improvement within your solution. For example, some of the key learnings from the evaluation of products and personnel often apply to the evaluation of programs and policies and vice versa. 0000004692 00000 n 2009). HEFCE indicated that impact should merit a 25% weighting within the REF (REF2014 2011b); however, this has been reduced for the 2014 REF to 20%, perhaps as a result of feedback and lobbying, for example, from the Russell Group and Million + group of Universities who called for impact to count for 15% (Russell Group 2009; Jump 2011) and following guidance from the expert panels undertaking the pilot exercise who suggested that during the 2014 REF, impact assessment would be in a developmental phase and that a lower weighting for impact would be appropriate with the expectation that this would be increased in subsequent assessments (REF2014 2010). 0000334705 00000 n Case studies are ideal for showcasing impact, but should they be used to critically evaluate impact? Metrics in themselves cannot convey the full impact; however, they are often viewed as powerful and unequivocal forms of evidence. The Payback Framework is possibly the most widely used and adapted model for impact assessment (Wooding et al. For systems to be able to capture a full range of systems, definitions and categories of impact need to be determined that can be incorporated into system development. Even where we can evidence changes and benefits linked to our research, understanding the causal relationship may be difficult. This atmosphere of excitement, arising from imaginative consideration transforms knowledge.. Research findings including outputs (e.g., presentations and publications), Communications and interactions with stakeholders and the wider public (emails, visits, workshops, media publicity, etc), Feedback from stakeholders and communication summaries (e.g., testimonials and altmetrics), Research developments (based on stakeholder input and discussions), Outcomes (e.g., commercial and cultural, citations), Impacts (changes, e.g., behavioural and economic). Clearly there is the possibility that the potential new drug will fail at any one of these phases but each phase can be classed as an interim impact of the original discovery work on route to the delivery of health benefits, but the time at which an impact assessment takes place will influence the degree of impact that has taken place. What are the methodologies and frameworks that have been employed globally to evaluate research impact and how do these compare? The growing trend for accountability within the university system is not limited to research and is mirrored in assessments of teaching quality, which now feed into evaluation of universities to ensure fee-paying students satisfaction. HEIs overview. Developing systems and taxonomies for capturing impact, 7. 0000007967 00000 n Findings from a Research Impact Pilot, Institutional Strategies for Capturing Socio-Economic Impact of Research, Journal of Higher Education Policy and Management, Introducing Productive Interactions in Social Impact Assessment, Measuring the Impact of Publicly Funded Research, Department of Education, Science and Training, Statement on the Research Excellence Framework Proposals, Handbook on the Theory and Practice of Program Evaluation, Policy and Practice Impacts of Research Funded by the Economic Social Research Council. n.d.). Here we address types of evidence that need to be captured to enable an overview of impact to be developed. Perhaps it is time for a generic guide based on types of impact rather than research discipline? This transdisciplinary way of thinking about evaluation provides a constant source of innovative ideas for improving how we evaluate. The current definition of health, formulated by the WHO, is no longer adequate for dealing with the new challenges in health care systems. 8. Over the past year, there have been a number of new posts created within universities, such as writing impact case studies, and a number of companies are now offering this as a contract service. This might describe support for and development of research with end users, public engagement and evidence of knowledge exchange, or a demonstration of change in public opinion as a result of research. By asking academics to consider the impact of the research they undertake and by reviewing and funding them accordingly, the result may be to compromise research by steering it away from the imaginative and creative quest for knowledge. One notable definition is provided by Scriven (1991) and later adopted by the American Evaluation Association (): "Evaluation is the systematic process to determine merit, worth, value, or . evaluation practice and systems that go beyond the criteria and their definitions. In this article, we draw on a broad range of examples with a focus on methods of evaluation for research impact within Higher Education Institutions (HEIs). 5. 4. It is now possible to use data-mining tools to extract specific data from narratives or unstructured data (Mugabushaka and Papazoglou 2012). We take a more focused look at the impact component of the UK Research Excellence Framework taking place in 2014 and some of the challenges to evaluating impact and the role that systems might play in the future for capturing the links between research and impact and the requirements we have for these systems. SIAMPI has been used within the Netherlands Institute for health Services Research (SIAMPI n.d.). (2007) adapted the terminology of the Payback Framework, developed for the health and biomedical sciences from benefit to impact when modifying the framework for the social sciences, arguing that the positive or negative nature of a change was subjective and can also change with time, as has commonly been highlighted with the drug thalidomide, which was introduced in the 1950s to help with, among other things, morning sickness but due to teratogenic effects, which resulted in birth defects, was withdrawn in the early 1960s. 0000342798 00000 n The Author 2013. The development of tools and systems for assisting with impact evaluation would be very valuable. In terms of research impact, organizations and stakeholders may be interested in specific aspects of impact, dependent on their focus. (2008), and Hanney and Gonzlez-Block (2011). Definition of Evaluation by Different Authors Tuckman: Evaluation is a process wherein the parts, processes, or outcomes of a programme are examined to see whether they are satisfactory, particularly with reference to the stated objectives of the programme our own expectations, or our own standards of excellence. 1.3. While defining the terminology used to understand impact and indicators will enable comparable data to be stored and shared between organizations, we would recommend that any categorization of impacts be flexible such that impacts arising from non-standard routes can be placed. Assessment for learning is ongoing, and requires deep involvement on the part of the learner in clarifying outcomes, monitoring on-going learning, collecting evidence and presenting evidence of learning to others.. Providing advice and guidance within specific disciplines is undoubtedly helpful. different meanings for different people in many different contexts. The quality and reliability of impact indicators will vary according to the impact we are trying to describe and link to research. This database of evidence needs to establish both where impact can be directly attributed to a piece of research as well as various contributions to impact made during the pathway. 0000002109 00000 n The aim of this study was to assess the accuracy of 3D rendering of the mandibular condylar region obtained from different semi-automatic segmentation methodology. This presents particular difficulties in research disciplines conducting basic research, such as pure mathematics, where the impact of research is unlikely to be foreseen. Other approaches to impact evaluation such as contribution analysis, process tracing, qualitative comparative analysis, and theory-based evaluation designs (e.g., Stern, Stame, Mayne, Forss, & Befani, 2012) do not necessarily employ explicit counterfactual logic for causal inference and do not introduce observation-based definitions. A university which fails in this respect has no reason for existence. %PDF-1.4 % The process of evaluation is dynamic and ongoing. Introduction, what is meant by impact? 2007). 0000328114 00000 n To enable research organizations including HEIs to monitor and manage their performance and understand and disseminate the contribution that they are making to local, national, and international communities. On the societal impact of publicly funded Circular Bioeconomy research in Europe, Devices of evaluation: Institutionalization and impactIntroduction to the special issue, The rocky road to translational science: An analysis of Clinical and Translational Science Awards, The nexus between research impact and sustainability assessment: From stakeholders perspective. These sometimes dissim- ilar views are due to the varied training and background of the writers in terms of their profession, concerned with different aspects of the education process. 0000007223 00000 n And also that people who are recognized as authors, understand their responsibility and accountability for what is being published. Differences between these two assessments include the removal of indicators of esteem and the addition of assessment of socio-economic research impact. It is perhaps worth noting that the expert panels, who assessed the pilot exercise for the REF, commented that the evidence provided by research institutes to demonstrate impact were a unique collection. 0000009507 00000 n A Review of International Practice, HM Treasury, Department for Education and Skills, Department of Trade and Industry, Yes, Research can Inform Health Policy; But can we Bridge the Do-Knowing its been Done Gap?, Council for Industry and Higher Education, UK Innovation Research Centre. Metrics have commonly been used as a measure of impact, for example, in terms of profit made, number of jobs provided, number of trained personnel recruited, number of visitors to an exhibition, number of items purchased, and so on. This petition was signed by 17,570 academics (52,409 academics were returned to the 2008 Research Assessment Exercise), including Nobel laureates and Fellows of the Royal Society (University and College Union 2011). The risk of relying on narratives to assess impact is that they often lack the evidence required to judge whether the research and impact are linked appropriately. To demonstrate to government, stakeholders, and the wider public the value of research. Authors from Asia, Europe, and Latin America provide a series of in-depth investigations into how concepts of . Downloadable! Many theorists, authors, research scholars, and practitioners have defined performance appraisal in a wide variety of ways. From 2014, research within UK universities and institutions will be assessed through the REF; this will replace the Research Assessment Exercise, which has been used to assess UK research since the 1980s.