|
In 2017 the government of Indonesia introduced a scoring system for all academics in the country that combines data from Scopus and Google Scholar with information submitted by Indonesian academics to track published papers, citations, and researchers’ h-index. These numbers are used to calculate a personal score that is taken into account when academics apply for research grants; a high score may also help with promotions and salary negotiations. The system generated controversy after national awards were given to the top scorers who were subsequently show to have achieved their scores by gaming the system.
The Chemical & Engineering News story on the controversy quotes LM author Ludo Waltman. Find the article here. Science also covered the story and quotes LM author Diana Hicks. Find that article here. The Russian Council for the Ethics of Scientific Publications of the Association of Scientific Editors and Publishers has published an open appeal to those who develop and implement quantitative indicators of publication activity. The document, inspired by the Leiden Manifesto, is signed by 21 eminent scholars. It proposes 12 principles tailored to the Russian context. Google translate generated a very nice translation into English, so the text should be accessible to all. Find it here.
With the posting of an Italian translation, we now have 20 translations of the Leiden Manifesto. Thanks go to Edoardo Milotti of Università di Trieste for producing this latest translation. The Leiden Manifesto has been translated into: Chinese, Traditional Chinese, Basque, Catalan, Czech, Danish, Finnish, French, German, Indonesian, Italian, Japanese, Korean, Persian, Portuguese, Russian, Serbian, Slovak, Spanish, and Swedish.
We would like to thank Surya Dalimunthe and Afrilya from the Indonesian Open Science Team for translating the Leiden Manifesto into Indonesian. The translation is now mounted on our translations page.
In light of rumours that only DORA signatories will have access to UKRI funding in future, Lizzie Gadd explains why Loughborough has chosen an alternative path up the ‘responsible metrics mountain’, and why she believes all ‘mountain-climbers’ should be equally supported and rewarded.
See the full blog post here Let’s move beyond the rhetoric: it’s time to change how we judge research
Five years ago, the Declaration on Research Assessment was a rallying point. It must now become a tool for fair evaluation, urges Stephen Curry in his call to begin changing research management practice: There have been many calls for something better, including the Leiden Manifesto and the UK report "The Metric Tide', both released in 2015. Like DORA, these have changed the tenor of discussions around researcher assessment and paved the way for change. It's time to shift from making declarations to finding solutions. . . . Stephen Curry, writing in Nature, 8 February 2018, vol. 554, p. 147  At the recent Nordic Workshop for Bibliometrics and Research Evaluation in Helsinki, Heidi Holst Madsen, from the Royal Danish Library and Marianne Gauffriau and Lorna Wildgaard from the University of Copenhagen presented an intriguing method of conveying the limits of bibliometric analysis to clients They propose a "consumer label" explaining how a bibliometric analysis fulfills or falls short on each of the 10 principles of the Leiden Manifesto. Here is their explanation with two sample consumer labels. Motivation In Denmark bibliometric evaluation has only very recently become a formal parameter of evaluation at Universities. The newly implemented model is inspired by the REF system, with the bibliometric evaluation informing departmental reflection as well as institutional evaluation. Unlike the universities of Bath or Bloomington Indiana there are no institutional principles for bibliometric analyses at the University of Copenhagen. We wondered therefore if the Leiden Manifesto could be implemented “bottom-up”, by bibliometricians and librarians, when doing bibliometric analyses. From experience we know that what we are asked for by university researchers and administration will not conform to the Leiden Manifesto principles, and while in an ideal world one would never deliver a bibliometric analysis that does not conform to all ten principles, in practice that can be difficult. Approach We developed two small case studies, within health sciences at the Copenhagen University Library Bibliometric Service, and investigated if the Leiden manifesto can be 1) used as a check list for ourselves to double-check just how responsible our analyses are, and 2) given to the consumer as a “consumer label” indicating the “quality of the ingredients” of the analysis. “Consumer labels” were developed for both cases, and discussed with the department head and the grant applicant, respectively. Please see the attached slides, that show the results of the Leiden Manifest as a consumer label for both cases. Summary of the main results In summary we found that in evaluation of our own practice bibliometricians could do more to ensure that those evaluated verify and legitimize the analysis (LM Principle 5). Further, in interpretation of the analysis, the consumer label mediated to the consumer in a very simple manner that not all research activities and publications are covered and how this can affect the results (Principle 3). This is a great benefit for us and consumer. Often we suspect the consumer skims the part of a bibliometric report concerning the limitations and restraints regarding how to interpret the bibliometric analysis. The consumer label was a useful tool when meeting with the consumer, as it helped us create a dialog about the division of responsibilities. It became clear to both consumer and bibliometrician that it is the responsibility of the client to supply the research mission and the bibliometrician to select appropriate indicators (Principle 2). Both clients were knowledgeable about bibliometric indicators but to systematically discuss a responsible use of the indicators was unfamiliar to them both Conclusions We acknowledge the potentials in the “consumer labels”, yet we also find that the implementation of the Leiden Manifesto becomes subjective. Specifically, the division of responsibilities is not described for any of the ten principles and the standard for fulfilling a principle is not clear. Next steps As a consumer label it is not intuitive – for example it may not be obvious to a consumer that the principle headlines come from the Leiden Manifesto, but the text is our evaluation. The Leiden Manifesto principles themselves are not self-explanatory, and it’s important that the consumers fully understand them. Further we have to question the reliability of the subjective evaluations of the Leiden Manifesto. If the evaluation is performed by a single bibliometrician, is the unreliability of this subjective interpretation too great? We do not have an answer yet. When we evaluated the analyses together, we did not immediately agree on the interpretation of all the principles and/or how the analyses conformed to it, pointing to a need for clarification of the principles. Thus our next steps are to analyze more use cases and investigate if the Leiden manifesto needs updating/validating to increase its longevity/usefulness. Particularly we are concerned with the division of responsibilities, and how the concepts within the principles be interpreted and operationalized. So the Leiden Manifesto is very much alive and kicking in Copenhagen and we look forward to working more with the application of the Leiden Manifesto over the coming months.  A paper has recently been published in Digital Library Perspecitves addressing concerns with applying the Leiden Manifesto. Find it here
The paper provides a critical discussion of the Leiden Manifesto for libraries already engaged in bibliometric practices. Full compliance with the Manifesto is time-consuming, expensive and requires a significant increase in bibliometric expertise with respect to both staffing and skill level. Despite these apparent disadvantages, it is recommended that all libraries embrace the Manifesto’s principles. The paper offers practical recommendations based on the work of the European Association for Research Libraries (LIBER) Working Group on Metrics. This work is in the beginning phase and summarizes literature on the topic, as well as the experiences of the members of the Working Group. The discussion reflects today's growing popularity of (quantitative) research assessment which is seen in enthusiasts introducing new metrics (i.e. altmetrics) and by critics demanding responsible metrics that increase objectivity and equity in evaluations. Carey Ming-Li Chen & Wen-Yau Cathy Lin from Taiwan have just published an article reflecting on lessons to be learned from DORA and the Leiden Manifesto. Although the article is written in Chinese, it includes a long summary in English which can be found at the end. The article compares the Leiden Manifesto and DORA: their frameworks, citation impact, and channels of dissemination. Comparing the document type and journals of citing works, the authors find that DORA has been emphasized by editorial board of journals, whereas the Leiden Manifesto has been cited by articles in scientometrics related fields or interdisciplinary mega journals. Since the Leiden Manifesto was published in Nature, it seems that it disseminated quickly and rapidly accrued citations. However, DORA was designed to be signed, to enable institutions to show that they are working against the impact factor. Nevertheless, as researchers in the scientometrics and research policy fields, the authors agree that indicators themselves are not evil, but we definitely should treat them correctly. The authors see the role of the Leiden Manifesto not limited to discussing the misuse of the impact factor or other indicators; in addition it aims to facilitate consensus between practitioner and stakeholders in their understanding of indicators and research assessment.
The paper is available here. |