Telehealth is defined by the Centers for Medicare and Medicaid Services as the use of telecommunications and information technology to provide access to health assessment, diagnosis, intervention, consultation, supervision, and information across distance.¹ The use of telehealth services during the COVID-19 pandemic has been unprecedented; from March to April 2020, case reports suggest that telehealth use went from less than 1 percent of visits,² to representing as much as 80 percent of visits, in places where COVID-19 prevalence was high.³ Although telehealth services have been available in the US for decades, adoption was still relatively uncommon pre-COVID-19.^{4, 5} The telehealth infrastructure has been in place in many health systems, but a number of barriers slowed down the uptake of the use of telehealth as a main mode of healthcare delivery. Barriers to use included limited insurance coverage, regulations regarding jurisdiction of licensure, and technical challenges for many providers and patients to offer and utilize these digitally mediated services.⁶ During COVID-19, however, telehealth emerged as a way to deliver socially-distanced care. In response, policymakers, payers, and providers eliminated almost all financial, regulatory, and technical barriers that hampered previous telehealth expansion initiatives.^7-9

Most policy, clinical, and e-health experts believe that while coverage policies and provider and consumer telehealth adoption levels may change following the end of the pandemic, the adoption trajectory of these technologies has been forever changed.^10-13 Therefore, assessing the provider and patient experience and the characteristics of telehealth care provided during COVID-19 is of great importance. Further, understanding the effectiveness of telehealth, and identifying technology interventions that allow for effective provider-patient interactions to improve access to care, reduce patient and provider burden, and inform decisions about the allocation of resources between in-person and telehealth services modes, is of interest.

Thus, the key decision dilemma is how to provide telehealth services rather than whether to provide telehealth services. In other words, we are seeking to identify which telehealth intervention works for which patient population in which setting and through which implementation strategy.

This topic was nominated by a member of the AHRQ Learning Health Systems (LHS) panel. Follow-up discussions by AHRQ with the nominator considered COVID-related changes in the use and coverage of telehealth and led to refinement in the scope to focus specifically on literature published since COVID-19. This report will focus on telehealth as remotely delivered and synchronous medical services (e.g., telephone, video visit) between a patient and a healthcare provider in an ambulatory setting (e.g., outpatient and community-based clinics) or emergency department (ED). We will consider studies of patients of all ages, including both pediatric and adult population. The primary end user of this report will be  healthcare systems, including hospitals, private practices, and other providers implementing telehealth services; the report will also be of interest to payers reimbursing providers for telehealth services, and patients engaging in or considering use of telehealth services.

The decision dilemma can be addressed by both quantitative and qualitative research designs, thus we are conducting a mixed-methods or integrative review.¹⁴ The state of evidence – having a variety of both quantitative and qualitative studies – also dictates that such an approach would be most useful and informative at this time. We will integrate quantitative evidence (using a systematic review) and qualitative evidence synthesis through a convergent segregated approach, where syntheses are conducted independently and simultaneously, and the quantitative and qualitative evidence is then integrated to provide a comprehensive picture of available evidence on telehealth throughout the pandemic.

Key Questions were made available for comment between June 17, 2021 and July 8, 2021. Sixteen sets of comments were received. The JHU EPC identified two common themes that would expand the scope of the review: (1) include literature prior to the era of COVID-19, and (2) include other types of virtual health beyond synchronous telehealth, such as wearable devices and apps. Each of these considerations had been discussed with the Stakeholders during the topic refinement phase; neither change was considered critical for this review. The LHS panel representative concurred with this decision.

The following are the Key Questions (KQ) to be addressed in the mixed methods review:

KQ 1.  What are the characteristics of patient, provider, and health systems using telehealth during the COVID-19 era, specifically:

1. What are the characteristics of patients (e.g., age, race/ethnicity, gender, socioeconomic status, education, geographic location (urban versus rural))?
2. What are the provider and health system characteristics (e.g., specialty, geographic location, private practice, hospital-based practice)?
3. How do the characteristics of patients, providers, and health systems differ between the first four months of the COVID-19 era versus the remainder of the COVID-19 era?

KQ 2.  What are the benefits and harms of telehealth during the COVID-19 era?

1. Does this vary by type of telehealth intervention (i.e., telephone, video visits)? 
2. Does this vary by patient characteristic (i.e., age, gender, race/ethnicity, type of clinical condition or health concern, geographic location)?
3. Does this vary by provider and health system characteristic (e.g., specialty, geographic location, private practice, hospital-based practice)?

KQ 3.  What is considered a successful telehealth intervention during the COVID-19 era:

1. From the patient or caregiver perspective?
2. From the provider perspective?
3. From the health system perspective?

KQ 4. What strategies have been used to implement telehealth interventions during the COVID-19 era?

1. What are the barriers and enablers of a successful telehealth strategy (e.g., setting, reimbursement, access to technology)?
   • From the patient or caregiver perspective?
   • From the provider perspective?
   • From the health system perspective?

We are addressing two Contextual Questions (CQ) about implementation as well as policy and reimbursement of telehealth.

CQ 1.   What are the costs of implementation and return on investment for telehealth during the COVID-19 era to the provider/healthcare system?

CQ 2.   What are the policy and reimbursement considerations for telehealth during the COVID-19 era?

1. How are these policy and reimbursement considerations for telehealth changing in the post-COVID-19 era (from March 2020, when the World Health Organization declared COVID-19 a pandemic to present); at the federal level (policies such as Medicare), state level (policies such as Medicaid), and by private insurance payers?
2. How do changes in reimbursement policies impact telehealth strategies?

ED = emergency department; IDS = integrated delivery system; IP = inpatient; KQ=Key Question; LHS = learning health system; OP = outpatient

A. Criteria for Inclusion/Exclusion of Studies in the Review

The criteria for inclusion and exclusion of studies for the mixed methods review will be based on the Key Questions and are described using the PICOTS framework in Table 1 (Population, Intervention, Comparison, Outcome, Timing, Setting). 

Table 1. PICOTS: Inclusion and exclusion criteria

* Studies that began before the era of COVID-19 (11 March 2020) and extend into the era of COVID-19 will be excluded unless they meet the following criteria: data from the pre and post COVID-19 era are stratified—the stratified data will be extracted; studies initiated as early as 1 January 2020 can be included if they are studies of telehealth in response to COVID-19.

^† To be eligible for inclusion as a qualitative study, the Sampling, data collection, and data analyses must be systematically conducted; data must be analyzed using methods of qualitative data analysis (such as thematic analysis).

CT = controlled trial; ED = emergency department; EHR = electronic health record; HEDIS = Healthcare Effectiveness Data and Information Set; KQ = key question(s); NA = not applicable, RCT = randomized controlled trial

Table 2. Proposed methods by key question.

CT = controlled trial; EHR = electronic health record; RCT = randomized controlled trial

Literature Search Strategies for Identification of Relevant Studies to Answer the Key Questions

Publication Date Range:

Searches will focus on studies conducted after the onset of the age of COVID-19 by initially limiting publication date from March 11, 2020 to present.

Literature searches will be updated while the draft report is posted for public comment. Literature identified during the updated search will be assessed in the same manner as all other studies considered for inclusion in the report.

Literature Databases:

We will search the following databases: PubMed, CINAHL, PsycINFO, and the Cochrane Central Register of Controlled Trials. We will develop a search strategy for PubMed, based on an analysis of the medical subject headings (MeSH) terms and text words of key articles identified a priori, and modify this for use in the other databases. The preliminary search strategies are included in Appendix C. Search strategies will be reviewed by an information specialist using the Peer Review of Electronic Search Strategies (PRESS) guidelines.¹⁵

We will hand search the reference lists of included articles and relevant systematic reviews. We will search clinicaltrials.gov to identify any relevant ongoing trials. Additionally, we will conduct targeted manual searches of selected telehealth-focused journals, will search grey literature on relevant websites (see Appendix D). We will not include pre-prints.

A Supplemental Evidence and Data for Systematic review (SEADS) portal will be available and a Federal Register Notice will be posted for this review.

Key Informant (Stakeholder) Input Strategies

We will engage the Stakeholders at two timepoints: (1) at the beginning of the project to provide input on inclusion and exclusion criteria, and potential information sources, and (2) at the end of the project to provide feedback on the integrative review process. We will compile key issues and themes noted by the Stakeholders and use those to inform our analysis of the qualitative and mixed-methods literature and the overall integration.

Screening, Data Abstraction and Data Management

We will use DistillerSR (Evidence Partners, 2010) to manage the screening process. DistillerSR is a web-based database management program that manages all levels of the review process. Unique citations identified by the search strategies will be screened in the following manner:

• Abstract screening: Two screeners will independently review abstracts, which will be excluded if both agree that the article meets one or more of the exclusion criteria listed in the above PICOTS table (Table 1). Differences between reviewers regarding abstract eligibility will be tracked and resolved through consensus adjudication.
• Full-text screening: Citations promoted on the basis of abstract review will undergo another independent review using full-text of the articles. The differences regarding article inclusion will again be tracked and resolved through consensus adjudication.

For the systematic review of quantitative evidence and the qualitative evidence synthesis, we will develop separate standardized forms for data extraction and pilot test them. Each study will undergo sequential data abstraction. All individuals involved in data abstraction will have experience in data abstraction for systematic review (junior reviewers) or will be experts in the area of telehealth (senior reviewers). The senior reviewer will confirm the first reviewer’s data abstraction for completeness and accuracy.

For all articles, reviewers will extract information based on the question addressed, generally to include study characteristics (e.g., study design, study period and follow-up, study location), characteristics of study participants (e.g., demographic, social, and clinical), type of telehealth service (telephone versus video visit), comparators (no service, in-person), clinical setting for providing the service (e.g., outpatient, ED, community based clinics, rural clinics), clinical specialty (e.g., adult primary care, pediatrics, neurology, surgery), and clinical conditions managed by the service (e.g., chronic condition management, behavioral health service). We will design data abstraction forms based on those used in past reviews to gather quantitative information on the effect of interventions (telehealth versus in-person) on outcomes of care (e.g., utilization of healthcare services such as ED or hospitalization following initial telehealth or in-person visit, clinical outcomes such as up to date labs, medication adherence, screening, vaccination, and management of chronic conditions). We will also add items from the Joanna Briggs Institute (JBI) data abstraction form, or similar tool, to collect qualitative data.¹⁶

In cases where study period begins prior to the COVID-19 era, we will extract data in the following manner:

• If data collection began between 1 January and 11 March 2020 and is in response to the COVID-19 crisis, we will abstract all data.
• If data collection began prior to the era of COVID-19 and extended into the era of COVID-19, we will extract data for COVID-19 era.
• If study includes comparison between COVID-19 era with prior COVID-19 we will collect all of the data.

If data are presented for both US-like and non-US-like countries, we will only extract data from US-like countries.

Assessment of Quality of Individual Studies

Paired investigators will assess studies independently for risk of bias. We will use the Cochrane Risk of Bias Tool, Version 2, for assessing the risk of bias of randomized controlled trials (RCTs)¹⁷; we will use the Cochrane Risk of Bias Assessment Tool for Non-Randomized Studies of Interventions (ROBINS-I) tool¹⁸ for non-randomized trials, including descriptive and observational studies. For qualitative and mixed-methods studies, reviewers will independently assess study quality using the JBI Checklist¹⁹ or similar tool. Descriptive studies will not undergo a risk of bias assessment.

Synthesis

Our approach to data synthesis will differ by Key Question.

Key Question 1

We will aggregate information on the types of telehealth interventions (e.g., telephone, video, etc.), and will present descriptive statistics of their use, by patients, providers, and health systems, during COVID-19. We will consider presenting user characteristics in different ways, such as a matrix of telehealth use by patient/provider characteristics. Further, we will compare the characteristics of telehealth use in the first four months of COVID-19 (March 2020 through July 2020) with telehealth use after the first four months.

Key Question 2

The question of benefits and harms of telehealth during COVID-19 will be addressed through a systematic review and a qualitative evidence synthesis.

For the systematic review, we will conduct qualitative synthesis and will perform meta-analysis where there are sufficient data to pool for analysis^{20, 21} We plan to address heterogeneity using subgroup analysis and meta-regression, if there are sufficient number of studies, or we will describe the heterogeneity qualitatively, if there are not. We will not combine clinically or methodologically diverse studies but, rather, we will describe the differences among the studies and population characteristics. If studies are not too diverse clinically or methodologically, we will evaluate the presence of statistical heterogeneity, using tests such as Cochran’s Q test and the I-squared statistic, to measure the magnitude of heterogeneity.^{22, 23} The 95 percent confidence interval for the I-squared statistic is intended to reflect the uncertainty in the estimate of the magnitude of heterogeneity. If statistical heterogeneity is attributable to one or two “outlier” studies, we will conduct sensitivity analyses by excluding these studies.

We will follow the JBI approach for the qualitative evidence synthesis.¹⁶ We plan to take a “best-fit” framework approach by developing a list of concepts and adopting, adapting or constructing a conceptual model regarding the perceived benefits and harms of telehealth. The framework would address the perceived benefits and harms of telehealth from the perspective of patients, providers, and health system. It would also present socioeconomic characteristics (e.g., age, gender, race, education, income, access to high-speed internet and advanced technology, and health literacy) and clinical conditions of patients (e.g., chronic condition, mental health) which would impact their perception of benefits and harms of telehealth. The framework would present characteristics of providers (e.g., primary care versus specialty care, academic versus community based) and health systems (e.g., urban versus rural and tertiary referral center versus community based) that would impact the perception of benefits and harms of telehealth services.

Key Question 3

We will address the question of what is considered a “successful” telehealth intervention through a qualitative evidence synthesis. We plan to create a matrix of users (e.g., patients, providers, and health systems), their characteristics, and their perspectives or expectations of a successful telehealth service. For instance, the matrix might present patients with low English proficiency defining the successful telehealth service as one in which they can engage a simultaneous translator in their communications with their providers. The matrix might also present primary care providers serving patients in a rural community defining the successful telehealth service as the one that they can complete through a smartphone application such as FaceTime.

Key Question 4

The question of implementation strategies for telehealth during COVID-19 will primarily be answered by descriptively summarizing the strategies, and barriers and facilitators identified. We will provide tables listing implementation strategies, barriers, and facilitators based on the characteristics of patient population (e.g., implementation strategies to provide telehealth services to low income racial minorities with limited access to high speed internet and low health literacy), characteristics of providers (e.g., implementation strategies for primary care, mental health, and specialty care providers), and  characteristics of health systems (e.g., implementation strategies in rural versus urban health systems and in low resource (federally qualified health centers and community based clinic) versus high resource setting (private practices and not for profit health systems)).     

Integration

For KQs 2 and 4 we will integrate the results from the systematic review and qualitative evidence synthesis. We plan to use a convergent segregated approach to synthesize and integrate the quantitative and qualitative data.¹⁶ In this approach, the syntheses of qualitative and quantitative studies are conducted separately and then these results are juxtaposed to determine how the findings complement each other. We may, if data and time allows, use an iterative approach, and take information gathered from quantitative sources to develop a matrix and map it to the qualitative data, which is better defined as a sequential approach.²⁴ Including this option will allow us to identify how the data from quantitative and qualitative sources complement one another (convergent), and identify where gaps between the two bodies of literature exist (sequential).

Grading the Strength of Evidence (SoE)

We will grade the body of evidence separately for quantitative and qualitative studies. For the systematic review of quantitative studies included in Key Questions 2 and 4, we will use the grading scheme recommended in the AHRQ Methods Guide for Effectiveness and Comparative Effectiveness Reviews (Methods Guide).²² For qualitative studies included in the qualitative evidence syntheses in Key Questions 2, 3, and 4, we will follow the GRADE-CERqual (Confidence in the Evidence from Reviews of Qualitative research)  approach.^25-31We have not pre-specified a subset of critical outcomes to be graded and will consider all outcomes. The list of outcomes reflects results of preliminary searching and input from Stakeholders and partners. We do not expect to find literature addressing each of these outcomes. The strength of evidence will not be graded for Key Question 1.

Assessing Applicability

We will consider elements of the PICOTS framework (Table 1) when evaluating the applicability of evidence to answer our Key Questions as recommended in the Methods Guide.²² This includes important population characteristics, characteristics of remotely delivered synchronous medical services, and settings that may cause heterogeneity and limit applicability of the findings.

Approach to Addressing Contextual Questions

The contextual Questions  will be answered by collating applicable data identified while addressing the Key Questions. For CQ2b we will also search current policy documents. Each CQ will also be informed by discussions with Stakeholders throughout the project.

1. Telemedicine. Centers for Medicare and Medicaid Services; 2020. Accessed on May 2021.
2. Telehealth Claim Lines Increase 8,336 Percent Nationally from April 2019 to April 2020. 2020. Accessed on October 16 2020.
3. Drees J. Led by COVID-19 surge, virtual visits will surpass 1B in 2020: report. 2020. Accessed on October 13 2020.
4. Barnett ML, Ray KN, Souza J, et al. Trends in Telemedicine Use in a Large Commercially Insured Population, 2005-2017. Jama. 2018 Nov 27;320(20):2147-9. doi: 10.1001/jama.2018.1235410.1001/jama.2018.12354. PMID: 30480716.
5. A Multilayered Analysis of Telehealth. 2019.
6. Lau J, Knudsen J, Jackson H, et al. Staying Connected In The COVID-19 Pandemic: Telehealth At The Largest Safety-Net System In The United States. Health affairs (Project Hope). 2020 Aug;39(8):1437-42. doi: 10.1377/hlthaff.2020.0090310.1377/hlthaff.2020.00903. Epub 2020 Jun 11. PMID: 32525705.
7. Hudman J, McDermott D, Shanosky N, et al. How Private Insurers Are Using Telehealth to Respond to the Pandemic. Accessed on October 16 2020.
8. Gardner C. Telehealth Innovation and Improvement Act of 2019. 2019.
9. Medicare Learning Network. Telehealth Services. 2020.
10. Shachar C, Engel J, Elwyn G. Implications for Telehealth in a Postpandemic Future: Regulatory and Privacy Issues. Jama. 2020 Jun 16;323(23):2375-6. doi: 10.1001/jama.2020.794310.1001/jama.2020.7943. PMID: 32421170.
11. Butler SM. After COVID-19: Thinking Differently About Running the Health Care System. Jama. 2020 Jun 23;323(24):2450-1. doi: 10.1001/jama.2020.848410.1001/jama.2020.8484. PMID: 32573658.
12. Totten AM, McDonagh MS, Wagner JH. AHRQ Methods for Effective Health Care. The Evidence Base for Telehealth: Reassurance in the Face of Rapid Expansion During the COVID-19 Pandemic. Rockville (MD): Agency for Healthcare Research and Quality (US); 2020.
13. A M, B W, S S. Telemedicine: What Should the Post- Pandemic Regulatory and Payment Landscape Look Like? ; 2020.
14. Lizarondo L, Stern C, Carrier J, et al. Chapter 8: Mixed methods systematic reviews. In: E A, Z M, eds. JBI Manual for Evidence Synthesis. JBI; 2020.
15. McGowan J, Sampson M, Salzwedel DM, et al. PRESS Peer Review of Electronic Search Strategies: 2015 Guideline Statement. Journal of clinical epidemiology. 2016 Jul;75:40-6. doi: 10.1016/j.jclinepi.2016.01.02110.1016/j.jclinepi.2016.01.021. Epub 2016 Mar 19. PMID: 27005575.
16. JBI Manaul for Evidence Synthesis. Joanna Briggs Institute; 2020. Accessed on June 2021.
17. Sterne JAC, Savovic J, Page MJ, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ (Clinical research ed). 2019 Aug 28;366:l4898. doi: 10.1136/bmj.l4898 10.1136/bmj.l4898. PMID: 31462531.
18. Sterne JA, Hernan MA, Reeves BC, et al. ROBINS-I: a tool for assessing risk of bias in non-randomised studies of interventions. BMJ (Clinical research ed). 2016 Oct 12;355:i4919. doi: 10.1136/bmj.i4919 10.1136/bmj.i4919. PMID: 27733354.
19. The Joanna Briggs Institute Critical Appraisal tools for use in JBI Systematic Reviews Checklist for Qualitative Research. Joanna Briggs Institute; 2017. Accessed on June 2021.
20. Castillo RC, Scharfstein DO, MacKenzie EJ. Observational studies in the era of randomized trials: finding the balance. The Journal of bone and joint surgery American volume. 2012 Jul 18;94 Suppl 1:112-7. doi: 10.2106/jbjs.l.00242 10.2106/JBJS.L.00242. PMID: 22810460.
21. Shikata S, Nakayama T, Noguchi Y, et al. Comparison of effects in randomized controlled trials with observational studies in digestive surgery. Annals of surgery. 2006 Nov;244(5):668-76. doi: 10.1097/01.sla.0000225356.04304.bc 10.1097/01.sla.0000225356.04304.bc. PMID: 17060757.
22. Methods Guide for Effectiveness and Comparative Effectiveness Reviews. AHRQ Publication No. 10(14)-EHC063-EF Agency for Healthcare Research and Quality. Rockville, MD: 2014. Effective Health Care Program
23. Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0. The Cochrane Collaboration; 2019. Accessed on September 10, 2019.
24. Harden A, Thomas J, Cargo M, et al. Cochrane Qualitative and Implementation Methods Group guidance series-paper 5: methods for integrating qualitative and implementation evidence within intervention effectiveness reviews. Journal of clinical epidemiology. 2017 May;97:70-8. doi: 10.1016/j.jclinepi.2017.11.02910.1016/j.jclinepi.2017.11.029. Epub 2017 Dec 11. PMID: 29242095.
25. Lewin S, Booth A, Glenton C, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings: introduction to the series. Implementation science : IS. 2018 Jan 25;13(Suppl 1):2. doi: 10.1186/s13012-017-0688-310.1186/s13012-017-0688-3. PMID: 29384079.
26. Colvin CJ, Garside R, Wainwright M, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 4: how to assess coherence. Implementation science : IS. 2018 Jan 25;13(Suppl 1):13. doi: 10.1186/s13012-017-0691-810.1186/s13012-017-0691-8. PMID: 29384081.
27. Munthe-Kaas H, Bohren MA, Glenton C, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 3: how to assess methodological limitations. Implementation science : IS. 2018 Jan 25;13(Suppl 1):9. doi: 10.1186/s13012-017-0690-910.1186/s13012-017-0690-9. PMID: 29384078.
28. Noyes J, Booth A, Lewin S, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 6: how to assess relevance of the data. Implementation science : IS. 2018 Jan 25;13(Suppl 1):4. doi: 10.1186/s13012-017-0693-610.1186/s13012-017-0693-6. PMID: 29384080.
29. Lewin S, Bohren M, Rashidian A, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 2: how to make an overall CERQual assessment of confidence and create a Summary of Qualitative Findings table. Implementation science : IS. 2018 Jan 25;13(Suppl 1):10. doi: 10.1186/s13012-017-0689-210.1186/s13012-017-0689-2. PMID: 29384082.
30. Glenton C, Carlsen B, Lewin S, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 5: how to assess adequacy of data. Implementation science : IS. 2018 Jan 25;13(Suppl 1):14. doi: 10.1186/s13012-017-0692-710.1186/s13012-017-0692-7. PMID: 29384077.
31. Booth A, Lewin S, Glenton C, et al. Applying GRADE-CERQual to qualitative evidence synthesis findings-paper 7: understanding the potential impacts of dissemination bias. Implementation science : IS. 2018 Jan 25;13(Suppl 1):12. doi: 10.1186/s13012-017-0694-510.1186/s13012-017-0694-5. PMID: 29384076.

Terms

Telehealth: remotely delivered and synchronous medical services (e.g., telephone, video visit) between a patient and a healthcare provider in an ambulatory setting (e.g., outpatient and community-based clinics) or emergency department (ED)

COVID-19 Pandemic: COVID-19 was declared a pandemic by the World Health Organization (WHO) on March 11, 2020

U.S.-like patient population: referring to patients served in a U.S.-like healthcare system, defined for this review as including countries in the list of Organization for Economic Cooperation and Development (OECD) nations with a World Bank classification at or above “upper-income.”

List of acronyms

The Agency for Healthcare Research and Quality (AHRQ) posted the Key Questions on the AHRQ Effective Health Care Website for public comment. The EPC refined and finalized them after reviewing of the public comments and seeking input from Key Informants and the Technical Expert Panel (TEP). This input is intended to ensure that the Key Questions are specific and relevant.

Due to the nature of this project, we are combining the Key Informants and Technical Experts into a group referred to as “Stakeholders” (see Methods section). Stakeholders include patients and caregivers, practicing clinicians, representatives from relevant professional and consumer organizations, purchasers of health care, and others with experience in making health care decisions, as well as clinical, content, and methodologic experts. We are engaging the Stakeholders throughout the project. During topic refinement, the Stakeholders provided input on the decisional dilemmas; defined populations, interventions, comparisons, or outcomes; identified particular studies or databases to search; and helped focus on Key Questions that will inform health care decisions. Stakeholders will also be engaged during the development of the protocol and throughout the review to provide input and guidance on the conduct of the review.

Stakeholders are expected to provide broad expertise and perspectives specific to the topic under development. Divergent and conflicting opinions are common and perceived as healthy scientific discourse that fosters a thoughtful, relevant review. Therefore, study questions, design, and methodological approaches do not necessarily represent the views of individual Stakeholders. Stakeholders provide information to the EPC to identify literature search strategies and suggest approaches to specific issues as requested by the EPC.

Stakeholders do not do analysis of any kind; neither do they contribute to the writing of the report. They do not review the report, except as given the opportunity to do so through the peer or public review mechanism.

Members of the Stakeholder group must disclose any financial conflicts of interest greater than $5,000 and any other relevant business or professional conflicts of interest. Because of their unique clinical or content expertise, individuals are invited to serve as Stakeholders and those who present with potential conflicts may be retained. The AHRQ TOO and the EPC work to balance, manage, or mitigate any potential conflicts of interest identified.

See above section on Key Informants describing the combination Key Informant and Technical Expert Panels for this review.

Peer reviewers are invited to provide written comments on the draft report based on their clinical, content, or methodological expertise. The EPC considers all peer review comments on the draft report in preparing the final report. Peer reviewers do not participate in writing or editing of the final report or other products. The final report does not necessarily represent the views of individual reviewers.

The EPC will complete a disposition of all peer review comments. The disposition of comments for systematic reviews and technical briefs will be published 3 months after publication of the evidence report.

Potential peer reviewers must disclose any financial conflicts of interest greater than $5,000 and any other relevant business or professional conflicts of interest. Invited peer reviewers with any financial conflict of interest greater than $5,000 will be disqualified from peer review. Peer reviewers who disclose potential business or professional conflicts of interest can submit comments on draft reports through the public comment mechanism.

EPC core team members must disclose any financial conflicts of interest greater than $1,000 and any other relevant business or professional conflicts of interest. Direct financial conflicts of interest that cumulatively total more than $1,000 will usually disqualify an EPC core team investigator.

This project was funded under Contract No. 75Q80120D00003 from the Agency for Healthcare Research and Quality, U.S. Department of Health and Human Services. The AHRQ Task Order Officer reviewed the EPC response to contract deliverables for adherence to contract requirements and quality. The authors of this report are responsible for its content. Statements in the report should not be construed as endorsement by either the Agency for Healthcare Research and Quality or the U.S. Department of Health and Human Services.

This protocol will be registered in the international prospective register of systematic reviews (PROSPERO)

Australia
Austria
Belgium
Canada
Chile
Czech Republic (HDI: 0.888)
Denmark
Estonia
Finland
France
Germany
Greece
Hungary
Ireland
Israel
Italy
Japan
Korea (South)
Latvia
Lithuania
Luxembourg
New Zealand
Norway
Poland
Portugal
Slovak Republic
Slovenia
Spain
Sweden
Switzerland
The Netherlands
UK
USA

*List of Organisation for Economic Cooperation and Development (OECD) nations excluding those with a World Bank classification below “upper-income.”

Appendix C: Detailed search strategies

Table C1: PubMed search strategy.

Table C2: CINAHL and PsycINFO

Table C3: Cochrane Database search

date limited, no reviews

Appendix D. Targeted Journals and Grey Literature Sources

Journals

Telemedicine and e-health

Journal of Telemedicine and Telecare

Nature – Digital Medicine

Journal of Medical Internet research

Journal of mhealth and u health

JMIR Medical Informatics

JMIR public health

JAMIA

Applied Clinical Informatics

BMC Medical informatics and decision making

Journal of Medical Systems

BMJ health and care informatics

Lancet digital health

Grey Literature

White Papers Archive

DiMe Research Projects

Changes to Telehealth Policy, Delivery, and Outcomes in Response to COVID-19

A White Paper Commentary on an AHRQ Evidence Report

The future of telemedicine Improving access and outcomes

Removing regulatory barriers to telehealth before and after COVID-19

Other resources/databases:

WHO COVID-19 database of primary studies: Global research on coronavirus disease (COVID-19)