Consumer labels for bibliometric analyses based on Leiden Manifesto

At the recent Nordic Workshop for Bibliometrics and Research Evaluation in Helsinki,  Heidi Holst Madsen, from the Royal Danish Library and Marianne Gauffriau and Lorna Wildgaard from the University of Copenhagen presented an intriguing method of conveying the limits of bibliometric analysis to clients  They propose a "consumer label" explaining how a bibliometric analysis fulfills or falls short on each of the 10 principles of the Leiden Manifesto.  Here is their explanation with two sample consumer labels.

Motivation
​In Denmark bibliometric evaluation has only very recently become a formal parameter of evaluation at Universities. The newly implemented model is inspired by the REF system, with the bibliometric evaluation informing departmental reflection as well as institutional evaluation. Unlike the universities of Bath or Bloomington Indiana there are no institutional principles for bibliometric analyses at the University of Copenhagen. We wondered therefore if the Leiden Manifesto could be implemented “bottom-up”, by bibliometricians and librarians, when doing bibliometric analyses. From experience we know that what we are asked for by university researchers and administration will not conform to the Leiden Manifesto principles, and while in an ideal world one would never deliver a bibliometric analysis that does not conform to all ten principles, in practice that can be difficult.

Approach
We developed two small case studies, within health sciences at the Copenhagen University Library Bibliometric Service, and investigated  if the Leiden manifesto can be 1) used as a check list for ourselves to double-check just how responsible our analyses are, and 2) given to the consumer as a “consumer label” indicating the “quality of the ingredients” of the analysis. “Consumer labels” were developed for both cases, and discussed with the department head and the grant applicant, respectively. Please see the attached slides, that show the results of the Leiden Manifest as a consumer label for both cases.

 Summary of the main results
In summary we found that in evaluation of our own practice bibliometricians could do more to ensure that those evaluated verify and legitimize the analysis (LM Principle 5). Further, in interpretation of the analysis, the consumer label mediated to the consumer in a very simple manner that not all research activities and publications are covered and how this can affect the results (Principle 3). This is a great benefit for us and consumer. Often we suspect the consumer skims the part of a bibliometric report concerning the limitations and restraints regarding how to interpret the bibliometric analysis. The consumer label was a useful tool when meeting with the consumer, as it helped us create a dialog about the division of responsibilities. It became clear to both consumer and bibliometrician that it is the responsibility of the client to supply the research mission and the bibliometrician to select appropriate indicators (Principle 2).  Both clients  were knowledgeable about bibliometric indicators but to systematically discuss a responsible use of the indicators was unfamiliar to them both
​
Conclusions
We acknowledge the potentials in the “consumer labels”, yet we also find that the implementation of the Leiden Manifesto becomes subjective.  Specifically, the division of responsibilities is not described for any of the ten
principles and the standard for fulfilling a principle is not clear.
 Next steps
As a consumer label it is not intuitive – for example it may not be obvious to a consumer that the principle headlines come from the Leiden Manifesto, but the text is our evaluation. The Leiden Manifesto principles themselves are not self-explanatory, and it’s important that the consumers fully understand them. Further we have to question the reliability of the subjective evaluations of the Leiden Manifesto. If the evaluation is performed by a single bibliometrician, is the unreliability of this subjective interpretation too great? We do not have an answer yet. When we evaluated the analyses together, we did not immediately agree on the interpretation of all the principles and/or how the analyses conformed to it, pointing to a need for clarification of the principles.
Thus our next steps are to analyze more use cases and  investigate if the Leiden manifesto needs updating/validating to increase its longevity/usefulness. Particularly we are concerned with the division of responsibilities, and how the concepts within the principles be interpreted and operationalized.
 So the Leiden Manifesto is very much alive and kicking in Copenhagen and we look forward to working more with the application of the Leiden Manifesto over the coming months.

---