From our colleagues at KU Leuven:
The use of metrics in assessing research(ers) has a long history and has often sparked controversy. Not infrequently, this was the result of a very strong faith in the power of numbers and the idea that quantitative data are objective and 'self-evident'. In addition, the wide availability and apparent simplicity of these data led to their great popularity. The calls for a more responsible use of metrics are to all extents justified. KU Leuven demurs from denouncing the value of metrics altogether or to merely replace existing metrics with new ones. Instead, the University considers the use of quantitative data in assessments to be relevant and justifiable, if both the providers and the users of these metrics know how to use them in a scientifically correct way. The Leiden Manifesto proved one of the most useful and relevant standards against which to set generally applicable practices and a simple checklist for use by evaluators and the evaluated. Both tools assist in assessing the degree to which metrics that are provided/used are in accordance with responsible behaviour. KU Leuven is now informing its staff and research evaluators to disseminate its nuanced policy on the use of metrics in research assessment.
Four key values constitute the pillars upon which assessments at KU Leuven are to be based: collective excellence, personal growth & innovation, leadership and Open Science. To endorse researchers to write their own narrative, KU Leuven introduced the biosketch in processes of hiring and promotion for senior academic staff in 2018.
Do metrics still have a place in this approach?
KU Leuven believes the answer to this question to be a full hearted “yes”.
KU Leuven is committed to creating a stimulating and appreciative environment for academics in which due attention is given to a diversity of profiles and personal trajectories. This commitment implies that assessments should appreciate and recognize of a wide range of activities and outcomes in the classical three academic spheres of research, education and engagement.
In November 2021, the Directorate General for Research and Innovation of the European Commission published a report entitled: Towards a reform of the research assessment system. The proposed reform builds on DORA, the Leiden Manifesto, and the Hong Kong Principles for assessing researchers. In January 2022, Nature endorsed the proposal in an editorial.
The European Commission consulted European and international stakeholders on how to facilitate and speed up reform so that the quality, performance and impact of research and researchers are assessed on the basis of more appropriate criteria and processes. The proposed way forward consists of a European agreement that would be signed by individual research funding organisations, research performing organisations and national/regional assessment authorities and agencies, as well as by their associations. The aim is for research and researchers to be evaluated based on their intrinsic merits and performance rather than on the number of publications and where these are published, promoting qualitative judgement with peer-review, supported by a more responsible use of quantitative indicators. The agreement would confirm the commitment of the signatories to changes, along commonly agreed objectives, principles and actions. It would offer a space for individual institutions to test changes, for mutual learning, and to more safely and efficiently engage in reforms.
Ismael Rafols, Sarah de Rijcke, Paul Wouters and Ludo Waltman wrote a letter to the University of Liverpool joining criticism of its use of metrics in deciding which faculty jobs are at risk. See the Nature article here.
Our thanks go out to Bilyana Yavrukova, Deputy Director of the biggest scientific library in Bulgaria - Sofia
University Library "St. Kliment Ohridski" who has translated the Leiden Manifesto into Bulgarian. It was also just
published in the Bulgarian journal Nauka: Лейденски манифест за научни метрики, Nauka, 2020, 2, p. 12-15.
There have been many discussions in Bulgaria concerning the evaluation of scientific publications and the work of Bulgarian researchers and there is a prototype Bulgarian citation index under development.
The University of Glasgow has developed a policy on the use of quantitative indicators in research evaluation that aligns with the principles of the Leiden Manifesto. In the statement, the university commits to:
1. Adopt assessment procedures that are evidence-based
2. Apply quantitative indicators responsibly by using a defined and balanced set of measures that are normalised by subject.
3. Declare the quantitative indicators used, and apply them fairly and consistently.
4. Evaluate researchers based on performance across different dimensions, with expectations set in advance and clearly communicated to researchers.
5. Undertake regular review of the quantitative indicators.
The statement on the use of quantitative indicators in research evaluation can be found here. It is located within a larger suite of policies governing research at the university that can be found here. The University is in the process of writing an annual statement on research culture in which they will detail their implementation plan.
Thank you Arş. Gör. Demet Işık and Prof. Dr. Özlem Gökkurt Demirtel for translating the Leiden Manifesto into Turkish. The translations page now provides a Turkish translation.
Thank you Eleonora Dagienė for translating the Leiden Manifesto into Lithuanian. The translation is now posted on our translations page.
In April, CWTS, Leiden will offer a short course addressing the key issues related to responsible uses of metrics in research management and research evaluation. In addition to theoretical, conceptual, and methodological approaches, a variety of case studies will be presented to illustrate issues related to responsible metrics. LM authors Sarah de Rijcke, Ludo Waltman, and Ismael Rafols, will be joined by Stephen Curry. Curry is one of the initiators of DORA and chair of the DORA steering committee.
Find more information here.
In 2017 the government of Indonesia introduced a scoring system for all academics in the country that combines data from Scopus and Google Scholar with information submitted by Indonesian academics to track published papers, citations, and researchers’ h-index. These numbers are used to calculate a personal score that is taken into account when academics apply for research grants; a high score may also help with promotions and salary negotiations. The system generated controversy after national awards were given to the top scorers who were subsequently show to have achieved their scores by gaming the system.
The Chemical & Engineering News story on the controversy quotes LM author Ludo Waltman. Find the article here.
Science also covered the story and quotes LM author Diana Hicks. Find that article here.
The Russian Council for the Ethics of Scientific Publications of the Association of Scientific Editors and Publishers has published an open appeal to those who develop and implement quantitative indicators of publication activity. The document, inspired by the Leiden Manifesto, is signed by 21 eminent scholars. It proposes 12 principles tailored to the Russian context. Google translate generated a very nice translation into English, so the text should be accessible to all. Find it here.