NIH - National Institutes of Health

11/19/2020 | News release | Distributed by Public on 11/19/2020 05:31

Should We Keep Meeting This Way?

Guest post by Dr. Bruce Reed, Deputy Director of the NIH Center for Scientific Review, originally released on the Review Matters blog

Dr. Bruce Reed, Deputy Director, CSR

How will study sections meet in the future? NIH peer review depends on robust meetings where groups of scientists, through vigorous discussion, identify the applications of highest merit. For the last 75 years, until last March, nearly all chartered review committee meetings were held in-person. Today, in response to the pandemic, 90% of all CSR review meetings are run as video ('Zoom') meetings. CSR is taking steps now so that when all options are back on the table, we can make informed choices about how best to convene review meetings.

Last round we obtained survey responses from 3,000 NIH reviewers, ratings by scientific review officers (SRO) of 230 review meetings, compiled quantitative data comparing in-person versus Zoom instances of over 275 meetings, analyzed rosters from those meetings, and also surveyed our support staff.

The data give no indication that the forced switch to Zoom has introduced major problems. Quality of review is our number one priority; 60% of reviewers and 85% of SROs say that overall quality has remained the same. By a large margin, reviewers report feeling equally able to speak and be heard in Zoom meetings compared to in-person meetings (see Figure). Reviewers, SROs, and support staff generally report the platform is easy to use, meetings are easy to manage, and technical problems are no more common or difficult to resolve than for in-person meetings.

Participation in Zoom Compared to In-Person Meetings

However, some of the survey results give us pause. Despite favorable ratings of Zoom meetings, reviewers and SROs prefer in-person meetings by very similar margins (43% to 31% for reviewers; 44% to 36% for SROs). Zoom meetings tend to run longer and the increased duration may explain some of that preference. Comparing Zoom to in-person, almost half of reviewers reported diminished attention, 51% perceived lower engagement, 30% contributed less, and 36% rated discussions as worse. In addition, many reviewers commented that they missed the social aspects of in-person meetings, the chances to network, to build collaborations, and to enjoy the comradery that in-person meetings foster.

The highly uneven impacts of COVID on scientists' daily lives make it difficult to discern how meeting format affects their ability/willingness to participate as reviewers. For many, family care responsibilities have increased during the pandemic and women are bearing a disproportionate load. We examined roster composition carefully to see if reviewer demographics have changed during the pandemic. So far, we see little change. Interpretation is complicated by the fact that reviewers are dealing with two large changes at once - video format, which may make it easier to participate, and COVID, which likely makes it harder.

We are all still learning. 'How to run better Zoom meetings' is a frequent topic of workshops, meetings, and staff conversations. Reviewers have reached out, for example, reminding us of the challenging home/work realities they are contending with. We understand that many reviewers must work from home and cannot completely absent themselves from family and household responsibilities. SROs generally are implementing more frequent breaks to accommodate this new reality. Even so, cameos by pets and children are not 'unprofessional', they're OK, perhaps even fun. We understand you might want to eat your breakfast with the video off. However, please come back, and mostly stay on screen because video seems to improve engagement. Hopefully, more breaks will help with Zoom fatigue, but one size does not fit all, and good communication between SROs and reviewers is vital.

Going forward CSR and OER plan to gather additional data from the February and March 2021 review meetings. We will consider those data, your comments, and input from our NIH colleagues when setting post-pandemic practices for study section meetings. There are many considerations: Travel is time-consuming, has environmental impacts, and is expensive; review should be an engaging and rewarding experience; meeting format may systematically make it easier, or harder for some scientists to participate; meetings vary considerably in size and so one solution may not fit all. And, hybrid solutions are on the table -for example, with some reviewers in the room and others joining by Zoom, or holding two meetings per year entirely by Zoom and one in person. The paramount consideration is quality of review. CSR is committed to maintaining the quality of review and improving it when possible.

We invite you to look at the details of our analyses thus far and tell us your ideas for how to improve Zoom review meetings. Please send your ideas to the Director of CSR's Office of Communications and Outreach, Kristin Kramer, or submit a comment.

Acknowledgments: Dr. Hope Cummings for survey work; Lia Fleming and Aditi Jain for quantitative analyses.