Measuring and Recognizing the Nuances of Research Impact
Highlights of ICOR Public Meeting #5, 10 July 2024
Please see the streaming video and chat record for rich details from speakers and meeting attendees.
Introduction
Research assessment is intrinsically linked to its associated journal metrics which determines the overall impact of a project. It’s well documented that this is a simplistic view of determining what impact actually is; however, the current incentives and reward system offer little room at the table for viable replacements. One way impact could be measured to better support the hiring, promotion, retention, and funding of scientists is by focusing on researchers’ commitment to open and collaborative research, and the effects of these practices. However, for open behaviours to contribute, compliment or even replace current impact metrics there are three critical problems requiring scholarly attention:
- How open and collaborative behaviours can be measured and finding tools to track and monitor these activities.
- How to explore and showcase evidence that open collaboration leads to higher quality and more impactful work, as well as possible unintended consequences;
- How to translate these desirable behaviours into incentives and rewards that are recognized by funders and academic institutions?
This meeting presented current initiatives that are employing research-on-research to explore these issues, and how resulting information can be used to better understand and improve funding practices and policies.
The first two talks were introduced by Iain Hrynaszkiewicz and covered the measurement of open science impacts, their causal mechanisms, and global efforts to monitor and build principles to support OS. The final talk covered the nuances of inferring broader impact and how to measure what is valued. The speakers shared the progress of their initiatives and encouraged the wider ICOR community to provide feedback and collaborate.
Open science impact pathways: A scoping review and a “living” handbook
Ioanna Grypari, Athena Research and Innovation Center, OpenAIRE and Vincent Traag, CWTS, Leiden University [slides; streaming video 6-27 min]
PathOS is a Horizon Europe project focused on modelling and quantifying the impacts of open science practice. Ioanna explained the aims and methodologies of their approach across academia, society, and the economy to enhance understanding and drive informed policy-making. She highlighted the project’s six case studies aimed at providing real-world, end-to-end stories of OS impact pathways and introduced their recently published Societal Impact of Open Science: A scoping review.
Vincent focused on the Open Science Indicator Handbook being developed, covering a number of indicators measuring open science and their academic, societal and economic impacts, and reproducibility. He explained the causal approach taken by the team in developing the handbook and ran through an illustrative simulation showing the extent and nature of the cause-and-effect relationship between open data and reproducibility. Importantly, the “living” handbook is being developed on Github to ensure it is versionable, interactive and open to community contributions. Vincent encouraged the community to provide the PathOS team with feedback and contribute to the handbook’s evolution.
Toward common principles for monitoring open science: the Open Science Monitoring Initiative
Laetitia Bracco, University de Lorraine [slides; streaming video 32-52 min]
In 2023, the G7 Science and Technology Ministers’ Communique announced its support for open science, with one of three priorities to use research-on-research to develop open science policies based on research results, including “inspiring a framework for open science monitoring”. Laetitia described existing national monitoring initiatives but explained there is no common understanding of OS monitoring and that a coordinated global approach is needed to develop alternatives to indicators based on publication. She then discussed her work at the French Open Science Monitor, where high-level principles were developed on how to include new objects for measurement into their monitoring system. This sparked the idea to harness international collaboration by gathering open science monitoring stakeholders and experts to work collaboratively on a set of global principles. After a UNESCO led consultation in December 2023, a draft “Principles of Open Science Monitoring” was produced on how to develop relevant, transparent, reproducible and responsible indicators.
These drafted principles led to the creation of the Open Science Monitoring Initiative, an international community of institutions and individuals aiming to encourage adoption of monitoring principles and promote their practical implementation. Laetitia ended her presentation with a call for the ICOR community to engage with the draft principles and to collaborate on OSMI (connecting via this form), as the intention is to make this a truly open and participatory global initiative.
Broader Impact: Collaboration on new measures of credit by all stakeholders
Kristen Ratan, Strategies for Open Science (Stratos) [slides; streaming video 60-75 min]
Stratos has been working with funders and institutions to better understand the meaning of research impact and how the traditional publication-based credit system can be enriched. Kristen described emerging credit measurements of research that can provide a more holistic view of contributions – e.g., openness of all outputs, timing of sharing, levels of collaboration and mentoring, research excellence, reuse metrics, equity and influence of policy and decisions. She noted that many funders and institutional leaders are currently reconsidering the sole use of publications to assess major inflection points in a researcher’s career, including HELIOS, ASAP and HHMI. This movement toward redistribution of credit requires collective input from the full spectrum of stakeholders to develop standards and principles that are reliable, clear, transparent, contextual, specific, and fair – with an understanding of how they will influence future research practices and policies.
Stratos’ work in this space would benefit from wider collaboration of the community, and Kristen ended by making a call for participation to all those interested in the broader effort (contact Kristen). This led to some excellent discussion and interest in working collectively in developing a more holistic approach to measuring research impact.
Summary
Overall, the meeting sparked much conversation during the group discussions (see meeting chat record); it was clear that those in the meeting care deeply about the next generation of impact measures. We encourage the ICOR community to share more case studies (via our “Submit your Project” form) on projects that are gathering evidence and implementing new measures and indicators of impact.
Please see the video recording for access to the full meeting.
ICOR Community members: please suggest topics or volunteer to host future public meetings on this Google Form.