Challenge Competition: Integrating Healthcare System Data with Systematic Review Findings

Challenge Competition: Announcement of AHRQ Challenge on Integrating Healthcare System Data With Systematic Review Findings

Dates

Phase 1 Submission Deadline on January 9, 2023 and Phase 2 Submission Deadline on July 10, 2023.

Supplementary Information

Problem Statement

The Agency for Healthcare Research and Quality (AHRQ), U.S. Department of Health and Human Services (HHS), is announcing a challenge competition to explore the feasibility, resources and infrastructure needed to integrate real world data from healthcare systems into systematic review findings. The statutory authority for this challenge competition is Section 105 of the America COMPETES Reauthorization Act of 2010.

AHRQ's Evidence-based Practice Center (EPC) Program produces systematic reviews which synthesize information from the peer reviewed literature and provide the state of the science on available healthcare technologies (such as pharmaceuticals and medical devices) as well as healthcare delivery strategies. The process of creating these reviews is stakeholder driven, methodologically rigorous, and transparent. Reviews are used to inform healthcare decisions, including recommendations in clinical practice guidelines as well as national coverage determinations by Medicare. AHRQ also supports healthcare systems in their efforts to improve the quality of care and optimize patient outcomes; systematic reviews are scoped to address issues of priority to healthcare systems. Yet, due to limitations in the literature base, EPC systematic reviews may be inconclusive or only represent a narrow patient population, making it difficult to generalize or implement the findings within heterogeneous healthcare systems.

Systematic reviews may also lack contextual details that can inform successful implementation. Improving healthcare delivery (and thus patient outcomes) often entails addressing issues beyond the benefits or harms of an intervention, traditionally the objective of a systematic review. Traditional reviews may not explain gaps in uptake or use of a clinical service and questions about how best to implement a given clinical service ( e.g., details around implementation of a service or intervention).

A recent EPC Program methods report ( https://effectivehealthcare.ahrq.gov/products/unpublished-health-data/methods-report ) articulates specific scenarios with examples of where healthcare system data may most effectively complement systematic reviews ( i.e., to improve the strength, applicability, and implementation of evidence). In one example, investigators at the Mayo Clinic found that published evidence on outcomes following total pancreatectomy was sparse, so they supplemented a meta-analysis of published studies with their own unpublished healthcare system data, which more than doubled the sample size and improved the strength of evidence available ( https://pubmed.ncbi.nlm.nih.gov/20681992/ ).

In another instance, secondary analyses of Veterans Affairs (VA) data ( https://pubmed.ncbi.nlm.nih.gov/35810550/ ) confirmed the applicability to the VA of findings from a published systematic review ( https://www.hsrd.research.va.gov/publications/esp/robot-gen-surg.cfm ).

The recent EPC methods report also outlines important limitations and considerations when using unpublished healthcare system data alongside systematic reviews, such as relevant limitations in data quality. However, the report did not address the necessary resources, skills, partnerships, and processes required to utilize healthcare system data alongside systematic reviews to strengthen the actionability of systematic reviews.

This Challenge therefore invites applicants to conduct an analyses of healthcare system data to supplement an existing AHRQ EPC Program systematic review. This will help AHRQ understand if and how sources of data and information outside of traditional systematic reviews, particularly from healthcare systems themselves, could be used alongside systematic reviews to improve decision making, healthcare delivery, and potentially patient outcomes.

Challenge Goal

The AHRQ EPC Program is interested in learning how analysis of real-world data collected by healthcare systems can be used in conjunction with findings from an AHRQ systematic review to inform healthcare decision-making in the context of a specific local setting.

The goal of this challenge is to explore and determine the feasibility, resources, and infrastructure needed to incorporate unpublished healthcare system data into systematic review findings. Ideally, these data will enable the healthcare system to make decisions about which practices to incorporate locally and how to overcome barriers to implement the evidence to improve clinical practice, healthcare system operations and, potentially, health outcomes. The winner and runner-up from Phase 2 will be posted on the AHRQ website.

Challenge Design

• All evidence reports (systematic reviews, rapid reviews, and technical briefs) published on the AHRQ website since January 2018 ( https://effectivehealthcare.ahrq.gov/about ) may be considered for this Challenge.

• Healthcare systems and other provider groups interested in implementing evidence into practice at their sites may apply.
• An organization may choose to address no more than 2 systematic reviews, submitting a separate proposal for each review.
• Teams should have expertise in the clinical topic, evidence-based practice, data analysis, and quality improvement.

This Challenge consists of two phases:

Phase 1: Proposal

Elicit written proposals on a topic for which an AHRQ evidence report (systematic review, rapid review or technical brief) has been published since January 2018 ( https://effectivehealthcare.ahrq.gov/about ) and that is relevant to a dilemma in the applicant's healthcare system. Each proposal is to be written in the form of a narrative that:

1. Explicitly states the rationale ( i.e., addressing evidence gaps to strengthen available evidence, examining applicability of findings to real-world patients) to complement findings from an EPC report with analysis of health system data, including a discussion of possible limitations of the analysis.

2. Develops and describes an analytic plan for use of healthcare system data [EHR data from an individual healthcare system or networks of healthcare systems (for example, PCORnet, Epic's Cosmos, etc.)].

3. Provides an approach for decision-making based on the results of the data analysis and the evidence report, and an evaluation of the decision-making process and results.

4. Describes potential challenges, barriers, and strategies to successfully complete the analysis.

5. Lists team members, their role, area of expertise and hours on project.

A total of 5 proposals will be selected as winners for Phase 1.

Phase 2: Analyses

Healthcare systems selected as winners in Phase 1 will be invited to provide a written narrative that:

1. Includes a complete analysis of internal real-world data and appraisal of the analysis for risk of bias using a formal tool such as the ROBINS I ( https://www.bmj.com/content/355/bmj.i4919 ).

2. Specifies how the findings from unpublished data support, refute, and/or otherwise complement findings from the published evidence examined in the systematic review. If the unpublished data conflicts with the AHRQ review's conclusions, discuss possible reasons for the discrepancy [ e.g., challenges with internal validity of healthcare system data analysis related to study design and methods used, or challenges with external validity with respect to population sub-groups (gender, race/ethnicity, multimorbidity) examined in the healthcare system data].

3. Describes how the unpublished data informed decision-making ( e.g., adapt, adopt, abandon, prioritize).

4. Reports on solutions to any barriers encountered to using healthcare system data alongside a published evidence review, including barriers with access to healthcare system data, interoperability of data sources, and data analysis.

5. Briefly describes potential approaches to implement or deimplement the evidence, including use of clinical advisories, clinical pathways, clinical decision support, or any other method. The plan should describe anticipated risks and barriers and strategies for successful implementation. Decisions made against uptake should be justified.

Timeline and Prize Amounts

AHRQ is hosting this challenge as a two-phase competition. All costs associated with developing and submitting proposals as well as conducting the analysis of real-world data will be the responsibility of the Challenge participant. Cash prizes will be awarded only after the projects are evaluated and determined acceptable at the end of Phase 2.

Timeline

• September 26, 2022-Challenge launch.
• January 9, 2023-Submissions for Phase 1 (written proposals) are due. AHRQ will complete the review of the proposals within 6-8 weeks of closing the announcement.
• March 10, 2023-AHRQ will announce the Phase 1 winners. Phase 2 of the Challenge will commence once the Phase 1 winners are announced and notified by March 10, 2023. The AHRQ team will schedule a live, virtual technical assistance webinar with all winners of Phase 1 to discuss scope of content, accessibility/compliance with Section 508, and address questions that the winners may have.
• March 10, 2023-Phase 2 participants will have at least 120 calendar days from notification to conduct and submit their analyses as described in their proposal(s). The deadline to submit the analysis is July 10, 2023.
• Fall 2023-The final winners of Phase 2 of the competition will be announced in October 2023.

Prize Amounts

The top five entries in phase one will be invited to participate in Phase 2. Upon completion of Phase 2, each of the top five entries will each receive cash awards of $50,000. Additionally, the first-place winner from Phase 2 will be awarded an additional $150,000 and the runner-up will be awarded an additional $100,000. The winner and runner-up from Phase 2 will be posted on the AHRQ website.

How To Enter the Challenge

Participants can register by visiting the Challenge.gov website ( https://www.challenge.gov ). Participants should carefully review challenge information and submission requirements on the website, including the intellectual property rules and assessment criteria. Participants are encouraged to follow the Challenge on Challenge.gov to obtain any updates and reminders of upcoming deadlines.

Submission Requirements

Phase 1

The submitted proposals must be written in US English and submitted using Challenge.gov no later than January 9, 2023. Applicants or applicant organizations shall submit no more than 2 proposals, and no proposal shall describe more than one topic. Each proposal will be no more than 2,000 words, double spaced, 11-point type size, with 1-inch margins. Include in proposals plans for meeting Section 508 accessibility standards.

Phase 2

Analyses shall be a journal article-style written narrative in U.S. English in no more than 4,000 words and submitted using the Challenge.gov website no later than July 10, 2023.

Review Process

All submissions will be reviewed by at least two subject matter experts from within or outside the federal government who will score the proposals based on the assessment criteria and provide a brief comment about the submission. The scores/comments on Phase 1 and Phase 2 submissions will be compiled, and a ranked summary provided to AHRQ staff. AHRQ will select winners based on quantitative and qualitative assessments.

Evaluation Criteria for Selecting Winning Applications

Assessment Criteria for Phase 1

1. Approach (40 points)

Does the proposal sufficiently describe:

a. the context and rationale for why the EPC report has been chosen, and how the analyses of unpublished data will complement the findings from the systematic review?

b. the data source and the analytic approach? Does the described analytic approach provide the right balance of feasibility, rigor, and innovation for the project?

2. Impact (40 points)

Does the proposal clearly and concisely describe:

a. potential routes for uptake of evidence within the healthcare system?

b. method to measure whether the uptake will have an impact on healthcare delivery, quality of care or patient outcomes?

c. anticipated risks, barriers, and strategies to successfully complete the project?

3. Team (20 points)

Does the team have the right combination of expertise to support the proposed technical approach?

Compliance (pass/fail)-Does the Phase 1 proposal adequately address required accessibility standards (Section 508)?

Assessment Criteria for Phase 2

Participants in Phase 2 may be disqualified if their submitted analyses deviate from their Phase 1winning proposals.

1. Analysis (40 points)

a. Are the description and discussion of the findings from the analysis comprehensive?

b. To what degree is the alignment/nonalignment between healthcare system data, the AHRQ systematic review, and the healthcare system's decision-making explained?

2. Description of the Process (40 points)

a. Are the risks, barriers, challenges encountered, solutions, and required infrastructure well described?

3. Future Plans (20 points)

a. Is the preliminary plan for implementation and evaluation appropriate and feasible?

Compliance (pass/fail)-Does the Phase 1 proposal adequately address required accessibility standards (Section 508)?

Eligibility Rules for Participating in the Challenge

To be eligible under this Challenge, an individual (whether participating singly or in a group) or entity:

1. Shall have registered ( Challenge.gov ) to participate in the Challenge.

2. Shall have complied with the rules set forth in this announcement for participation in this Challenge.

3. Shall be incorporated and maintain a primary place of business in the United States (in the case of a private entity), and in the case of an individual, whether participating singly or in a group, shall be a citizen or permanent resident of the United States.

4. May not be a Federal entity or Federal employee acting within the scope of their employment. (All Federal employees should consult with their agency Ethics Official to determine whether the federal ethics rules will limit or prohibit the acceptance of a prize).

5. May not be an employee of AHRQ or any other company, organization, or individual involved with the design, production, execution, judging, or distribution of the Challenge, or their immediate family (spouse, parents and step-parents, siblings and step-siblings, and children and step-children), or household members (people who share the same residence at least 3 months out of the year).

6. May not use Federal funds from a grant to develop Challenge applications unless consistent with the purpose of the grant award.

7. May not use Federal funds from a contract to develop Challenge applications or to fund efforts in support of a Challenge submission.

8. Shall not be deemed ineligible because the individual or entity used Federal facilities or consulted with Federal employees during a competition if the facilities and employees are made equitably available to all individuals and entities participating in the competition.

9. Shall not be required to purchase liability insurance as a condition of participation in this competition.

Additional Rules of Participation

By participating in this Challenge, each individual (whether participating singly or in a group) or entity:

1. Agrees to follow all applicable Federal, State, and local laws, regulations, and policies.

2. Agrees to comply with all terms and conditions of participation in this Challenge.

3. Agrees that the submission will not use HHS or AHRQ logos or official seals, except as authorized by HHS or AHRQ. Notwithstanding this authorized use of AHRQ/HHS branding, participants will not claim endorsement by AHRQ/HHS.

4. Understands that all materials submitted to AHRQ as part of a submission become AHRQ records. Any confidential commercial or financial information contained in a submission must be clearly designated as such at the time of submission.

5. Agrees that a winning submission may only be announced by AHRQ and in a time, place, and manner determined by AHRQ in its sole discretion. Winners will be permitted to publicize and publish their winning submissions in accordance with instructions provided by AHRQ. The winner and runner-up from Phase 2 will be posted on the AHRQ website.

6. Agrees that the submission must not infringe upon copyright or any other rights of any third party.

7. Agrees to assume any and all risks and waive claims against the Federal Government and its related entities, except in the case of willful misconduct, for any injury, death, damage, or loss of property, revenue, or profits, whether direct, indirect, or consequential, arising from participation in this prize contest, whether the injury, death, damage, or loss arises through negligence or otherwise.

8. Agrees to indemnify the Federal Government against third-party claims for damages arising from or related to Challenge activities.

9. Understands that circulation of findings could be worldwide, and that the Federal Government will not compensate the participants for any use; winners shall receive a one-time cash prize as set forth in this announcement. The winner and runner-up from Phase 2 will be posted on the AHRQ website.

10. Understands that AHRQ reserves the right to cancel, suspend, and/or modify this prize contest, or any part of it, for any reason, in AHRQ's sole discretion. AHRQ also reserves the right not to award any prizes if no entries are deemed worthy.

11. Understands that AHRQ will not select a winner that is named on the Excluded Parties List System (EPLS).

Intellectual Property (IP) Rights

1. Each participant retains title and full ownership in and to their submission. Participants expressly reserve all intellectual property rights not expressly granted.

2. By participating in the Challenge, each participant (whether participating singly or in a group) acknowledges that he or she is the sole author or owner of, or has a right to use, any copyrightable works that the submission comprises, that the works are wholly original with the participant (or is an improved version of an existing work that the participant has sufficient rights to use and improve), and that the submission does not infringe any copyright or any other rights of any third party of which participant is aware.

3. In addition, each participant (whether participating singly or in a group) grants to the U.S. Government a paid-up, nonexclusive, royalty-free, irrevocable worldwide license in perpetuity, and the right to reproduce, publish, post, link to, share, display publicly (on the web or elsewhere) and prepare derivative works, including the right to authorize others to do so on behalf of the U.S. Government.

4. Each participant must clearly delineate any intellectual property and/or confidential commercial information contained in a submission that the participant wishes to protect as proprietary data, in accordance with Additional Rules of Participation No. 4.

5. If the submission includes any third-party works (such as third-party content or open-source code), the participant must be able to provide, upon request, documentation of all appropriate licenses and releases for use of such third-party works. If the participant cannot provide documentation of all required licenses and releases, AHRQ reserves the right, in its sole discretion, to disqualify the submission.

[FR Doc. 2022-20703 Filed 9-23-22; 8:45 am]

BILLING CODE 4160-90-P