Fostering Incentives & Rewards for Open Practices
Highlights of ICOR Public Meeting 2, July 20, 2023
Register for the next Public Meeting Sep 28th, 10:30am-12:00pm ET, with presentations and discussion on the intersection between open and equitable scholarship. Details to follow.
Streaming video and chat record of this meeting
Meeting context and ICOR updates
Elusive goals
Since ICOR’s inception in 2020 we’ve seen remarkable innovations being developed and tested by community organizations – many of whom are featured in the ICOR website and in our meetings – determined to forge open collaboration in the research establishment. Yet even when inspiring success stories unfold, such as globally shared resources during the pandemic, there remains a hypercompetitive culture due largely to an intractable reward system based on publication metrics. And even as we continue to witness the rapid growth of funded transdisciplinary and multi-institutional initiatives, the desired outcomes remain elusive when uncoupled from coordinated incentives for researchers to share outcomes early, openly and collegially.
The focus of our second public meeting on July 20 was to address this issue by showcasing strategies for reform and their potential impact. Following are highlights of the topics, speakers, and discussion points (see lively Zoom chat record with many useful links) on incentives and rewards, plus a couple updates on related events (see also streaming video).
ICOR’s community project library – Kristen Ratan, Moderator and ICOR Co-Founder/Director
ICOR has begun to build an evidence base, a library of standards and best practices, and community projects focused on open and collaborative research. Given the fractured landscape in this arena, it is critical to unite and formalize the process, to match you, your innovations and efforts with research projects that are ripe to be studied. There has been a tremendous response in the past months from those who share this mission, by clicking on the I’m Interested button to receive ICOR updates or the Submit your Project button as a means of showcasing your Solution in the ICOR Project Library. We plan to work closely with groups pushing agendas similar to ours, some of which are described below.
“Accelerating the Adoption of Open Science Summit” convened by NASA & CERN (July 14-16) – Greg Tananbaum, Director, Open Research Funders Group (ORFG)
In celebration of 2023 as the year of open science, NASA joined with CERN to organize this summit, recently held in Geneva. The goal of the gathering was to exchange experience, ideas and expertise toward promoting open science policies and practices. Each day started with a public plenary offering insight into a specific area of the field such as open software and open data, followed by small clinics designed to identify strategies, challenges and commonalities that can be pursued across geographies and communities.
We can expect a summary statement from CERN/NASA that defines participants’ commitment to accelerating the transition to a more open and participatory ecosystem, as well as a mapping exercise of efforts and priorities that are currently underway. Much more information about plans and eventual statements from CERN and NASA can be found on the dedicated summit website linked above.
Fostering Incentives & Rewards
NASEM Roundtable on Aligning Incentives & UCSF efforts – Keith Yamamoto, Chair of the Roundtable and Vice Chancellor for Science Policy and Strategy, UCSF
The National Academies organized a Roundtable (described here) involving a substantial group of senior leaders from academia, planning agencies, foundations, and industry to develop actionable guidance and resources that align research incentives with open research and sharing practices (recently Aaron O’Shea joined Keith as co-chair of the Roundtable). In 2021 NASEM released a Toolkit for fostering open science practices, a collection of documents that state the imperative for open science using clear language for use by stakeholders, as well as resources such as templated grant applications, job postings, worksheets, success stories and recommendations for an array of research outputs. As part of the “coalition of the engaged”, the HELIOS effort was launched as described below by the program manager. As well, multisector workshops are being staged to consider how research is being impacted by the Nelson OSTP memo, such as the requirement for free and immediate access to federally funded research.
Keith also discussed the framework being developed and socialized at UCSF proposing (1) the removal of disincentives for open science and false metrics currently used to define research quality, and (2) the inclusion of positive incentives for open behaviors. All deans of all schools at UCSF are currently on board and the framework document is being reviewed by the academic senate to approve and finalize its form.
Arcadia Science: Innovative approaches to reforming research incentives and rewards – Prachee Avasthi, Co-Founder and Chief Scientific Officer, Arcadia Science [see slides]
Arcadia Science’s goal is to re-envision science by how research is done and communicated. The goal of its publishing experiment is to publish early and often. Modular research products including articles are being released publicly throughout the research lifecycle, if cleared for IP, to share their usefulness and to receive public feedback on the work (which Arcadia researchers contribute to on community preprints in kind). Best practices are being followed such as using persistent identifiers, making research products searchable and discoverable via repositories, and using CRediT taxonomy to acknowledge all contributors and their roles. The central focus is on outputs’ reuse and utility rather than publication venue and metrics.
As a for-profit company, Arcadia is, however, facing some challenges such as the need to develop infrastructure and personnel support to enable and sustain researchers’ sharing mandates; and the concerns of scientists about releasing research products that are complete or of high quality. Approaches Arcadia has tried and tested include setting sharing goals and taking on responsibility for doing so; sharing publication metrics with managers who weigh in during performance evaluation for improved accountability; using a dedicated publication team (doubling it to eliminate bottlenecks) for support, implementation, accountability and project management; and engaging in team activities to address backlogs and celebrate new releases.
Research assessment at the Howard Hughes Medical Institute – Anna Hatch, Program Officer, Open Science Strategy, HHMI [slides forthcoming]
Anna highlighted the importance and role of research assessment (see slides). As an organization, HHMI is expanding their view of scientific excellence to take into account how their values and priorities are integrated into research practices and behaviors. As a signatory of the Declaration on Research Assessment (DORA) HHMI has begun to de-emphasize journal names in researcher assessment. They are currently socializing the concept and have implemented the practice of removing journal names from HHMI science meetings, using PMIDs instead where available. In addition, they will develop a citation style with Zotero that scientists can use to help generate their bibliographies. Their holistic approach to researcher assessment aims to make all varieties of contribution – such as developing new tools and protocols, sharing outputs, pushing new directions, mentoring, training – visible and valuable in the research culture.
An internal group is working on setting expectations, revising application materials, and developing assessment guidelines for their scientists.
Higher Education Leadership Initiative on Open Science (HELIOS) – Caitlin Carter, Program Manager, Helios [see slides]
HELIOS emerged from the NASEM Roundtable and includes (currently) 95 institutions whose presidents or provosts are committed to prioritize open scholarship to strategic priority and to share with each other their initiatives. It is aligned with the Open Research Funders Group (ORFG). The core idea is to align values and reward actions such as collaboration, inclusion, transparency including FAIR data sharing, and open access publication. Of the four HELIOS subgroups, the Institutional and Departmental Policy Working Group is developing a collective action plan for embedding open scholarship considerations within hiring reappointment, promotion and tenure guidelines while respecting institutional and disciplinary differences. They’ve issued a first public brief, RPT (Reappointment, Promotion, Tenure) Joint Statement, outlining activities that institutions can engage in to advance open scholarship.
The policy working group along with members of the NASEM also released Meeting New Federal Requirements for Research Access: Opportunities for Leadership Action, that maps public access compliance opportunities based on the policies that have sprung from the Nelson OSTP memo. HELIOS has begun to work with professional societies such as the American Anthropological Association and the Association for Psychological Science to address discipline-specific changes that are needed for policy change. So while HELIOS is pushing top-down action, there is also a grass roots and collaborative approach to ensure that multiple sectors are represented in contextually informed documentation and resources.
Stanford Program on Research, Rigor and Reproducibility (SPORR)- Steve Goodman, Director, and Mario Malički, Associate Director of SPORR, Stanford University [see slides]
SPORR is a project at the Stanford School of Medicine, funded by a Clinical and Translational Science Awards (CTSA) grant and developed as a model for other Stanford schools and departments. The program provides support services for education and training, monitoring and accountability, and implementation of open scholarship practices to both wet and dry lab scientists as well as students. The training programs focus both on rigor and reproducibility – i.e., quality and openness of research – in probably a 60/40 ratio, with rigor on top. Training programs include journal clubs, rounds of colloquia, and the developing “clinical research observatory” whereby faculty and trainees learn about clinical research programs. There is training of rigor and reproducibility (R&R) representatives that will mainly be graduate students and postdocs embedded in labs and research teams.
For monitoring and accountability, SPORR is developing metrics for the % of clinical trials that follow transparency indicators such as results reported in registry and/or journals, open access, and prospectively registered trials. Slides 12 & 13 show templates for their dashboard reporting reporting the data and study-specific report cards. They are conducting a school-wide survey of data management plans and auditing manuscripts for open science that they will repeat periodically.
In partnership with CORES (Center for Open Reproducible Science) in the School of Engineering, SPORR developed a template (slides 15&16) for CVs that includes: (1) ORCIDs; (2) R&R annotated bibliography of research outputs; (3) percentage summaries of outputs in transparency categories; (4) narratives on contributions to science (per NIH biosketch); (5) editorial services such as open reviews, preprint and post-publication reviews, journal editorship including number of peer reviews, etc. (6) teaching and presentations with annotated links; and (7) a summary statement of R&R applications with content suggestions (see slides 17&18 for a sample R&R statement).
Finally, they’ve begun rewarding those that engage in and promote open scholarship ($1,000) – see this SPORR Awards page for 2023 winners and their projects, plus requirements to submit. A great example of a winning entry was an online open source lab manual that describes, among other topics, authorship guidelines, creation of reproducible workflows, and data management.
SPORR is planning to organize a national colloquium on R&R education in January 2024 that will be open to all.
There will also be a report soon on the CTSA wide survey on RR courses across USA; the project call can be seen here: https://med.stanford.edu/spectrum/about-spectrum/news/stanford-launches-ctsa-wide-survey-on-rigor-and-reproducibility.html
Register for the next ICOR Public Meeting on September 28th, 2023, 10:30am-12:00pm ET, with presentations and discussion on the intersection between open and equitable scholarship. Details to follow.
Feature image by Amy Shamblen on Unsplash