Did the agency have senior staff members with the authority, staff, and budget to build and use evidence to inform the agency’s major policy and program decisions in FY20?

1.1 Did the agency have a senior leader with the budget and staff to serve as the agency’s Evaluation Officer (or equivalent)? (Example: Evidence Act 313)

• The Director of the Office of Performance and Evaluation (OPE) serves as the Administration for Community Living (ACL) evaluation officer. OPE, which oversees the agency’s performance and evaluation work, has six full time staff positions and three full-time onsite contractors. In FY20 it had a budget of approximately $10.1 million. The Director of OPE has the education, skill, and experience to meet the Evaluation Officer requirements listed in the Evidence Act and routinely gauges the coverage, quality, methods, consistency, effectiveness, independence, and balance of the portfolio of evaluations, policy research, and ongoing evaluation activities of the agency and assesses agency capacity to support the development and use of evaluation. The Director is also the designated ACL Performance Officer.

1.2 Did the agency have a senior leader with the budget and staff to serve as the agency’s Chief Data Officer (or equivalent)? (Example: Evidence Act 202(e))

• The Director of the Office of Performance and Evaluation (OPE) serves as the Administration for Community Living’s Chief Data Officer. OPE, which oversees the agency’s performance and evaluation work, has six full time staff positions and three full-time onsite contractors. In FY20 it had a budget of approximately $10.1 million. The Director of OPE leads ACL’s Data Governance Body, including facilitating collaborative activities among the numerous actors with responsibilities and needs for data within the agency and has demonstrated training and experience in data management, governance, collection, analysis, protection, use, and dissemination and fulfills the aspects of this role which are relevant to ACL. These include coordinating with ACL’s CIO and Chief Privacy Officer on use, protection, dissemination, and generation of data to ensure that the data needs of the agency are met; ensuring that agency data conform with data management best practices; engaging agency employees, the public, and contractors in using public data assets; and encouraging collaborative approaches on improving data use. The Director of OPE serves as the agency liaison to other federal entities through, for example, serving as the ACL representative to the HHS data council, and serving on the Federal Interagency Council on Evaluation Policy as well as the HHS Data Governance Board.

1.3 Did the agency have a governance structure to coordinate the activities of its evaluation officer, chief data officer, statistical officer, and other related officials in order to inform policy decisions and evaluate the agency’s major programs?

• The Director of ACL’s Office of Performance and Evaluation serves the functions of evaluation officer, chief data officer, and performance officer. In order to coordinate activities relevant to these positions, the OPE Directors and staff coordinate the support, improvement, and evaluation of agency programs through implementation of an agency performance strategy, learning agenda, annual agency wide evaluation plan, and additional long range and evaluation plans for the Administration on Aging (in development) and the National Institute for Disability, Independent Living, and Rehabilitation Research. The structure requires semi-annual meetings with ACL leadership and management staff and annual consultation with all program managers. In FY19 ACL instituted a council to improve ACL’s data governance and quality, including the development of improved processes and standards for defining, collecting, reviewing, certifying, analyzing, and presenting data that ACL collects through its evaluations, grant reporting, and other administrative data collections. Taken together, this robust governance structure ensures cohesive collection and use of evidence across ACL regarding program performance, evaluation, and improvement and to ensure that data are gathered, processed, and curated so as to produce evidence that program staff and agency leadership use for program and operational improvement. As an operating division without a statistical unit, ACL does not have a statistical officer.

Did the agency have an evaluation policy, evaluation plan, and learning agenda (evidence-building plan), and did it publicly release the findings of all completed program evaluations in FY20?

2.1 Did the agency have an agency-wide evaluation policy? (Example: Evidence Act 313(d))

• ACL’s public evaluation policy confirms ACL’s commitment to conducting evaluations and using evidence from evaluations to inform policy and practice. ACL seeks to promote rigor, relevance, transparency, independence, and ethics in the conduct of evaluations. The policy addresses each of these principles.

2.2 Did the agency have an agency-wide evaluation plan? (Example: Evidence Act 312(b))

• ACL’s agency-wide evaluation plan was submitted to the Department of Health and Human Services (HHS) in support of HHS’ requirement to submit an annual evaluation plan to OMB in conjunction with its Agency Performance Plan. ACL’s annual evaluation plan includes the evaluation activities the agency plans related to the learning agenda and any other “significant” evaluation, such as those required by statute. The plan describes the systematic collection and analysis of information about the characteristics and outcomes of programs, projects, and processes as a basis for judgments, to improve effectiveness, and/or inform decision-makers about current and future activities.

2.3 Did the agency have a learning agenda (evidence-building plan) and did the learning agenda describe the agency’s process for engaging stakeholders including, but not limited to the general public, state and local governments, and researchers/academics in the development of that agenda? (Example: Evidence Act 312)

• Based on the learning agenda approach that ACL adopted in 2018, ACL published a learning agenda, in FY20. In developing the plan, ACL engaged stakeholders through meetings with program staff and grantees as required under OMB M-19-23. Additional meetings with stakeholder groups, such as through conference sessions, were put on hold for 2020 due to COVID-19 travel restrictions.

2.4 Did the agency publicly release all completed program evaluations?

• ACL releases all evaluation reports as well as interim information such as issue briefs, webinar recordings, and factsheets based on data from its evaluation and evidence building activities.

2.5 What is the coverage, quality, methods, effectiveness, and independence of the agency’s evaluation, research, and analysis efforts? (Example: Evidence Act 315, subchapter II (c)(3)(9))

• Staff from the Office of Performance and Evaluation (OPE) played an active role HHS’s capacity assessment efforts serving on the Capacity Assessment Subcommittee of the HHS Evidence and Evaluation Council. ACL’s self-assessment results were provided to HHS to support HHS’ ability to submit the required information to OMB. ACL’s self-assessment results, which provided information about planning and implementing evaluation activities, disseminating best practices and findings, and incorporating employee views and feedback; and carrying out capacity-building activities in order to use evaluation research and analysis approaches and data in the day-to-day operations will be reviewed by the ACL Data Council in order to develop a capacity building plan.

2.6 Did the agency use rigorous evaluation methods, including random assignment studies, for research and evaluation purposes?

• ACL typically funds evaluation design contracts, such as those for the Older Americans Act Title VI Tribal Grants Program evaluation and the Long Term Care Ombudsman Evaluation, that are used to determine the most rigorous evaluation approach that is feasible given the structure of a particular program. While the Ombudsman program is full coverage programs, where comparison groups are not possible, ACL most frequently uses propensity score matching to identify comparison group members. This was the case for the Older Americans Act Nutrition Services Program and National Family Caregivers Support Program evaluations and the Wellness Prospective Evaluation Final Report conducted by CMS in partnership with ACL and published in January 2019.
• ACL’s  National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) funds the largest percentage of ACL’s RCTs. Of their 718 research projects being conducted by grantees, 23% (163/718) are employing a randomized clinical trial (RCT) or “true experimental” design. To ensure research quality, NIDILRR adheres to strict peer reviewer evaluation criteria that are used in the grant award process (see part (c) for details on rigor of research projects and part (d) for details on the design of research projects). In addition, ACL’s evaluation policy states that “In assessing the effects of programs or services, ACL evaluations will use methods that isolate to the greatest extent possible the impacts of the programs or services from other influences such as trends over time, geographic variation, or pre-existing differences between participants and non-participants. For such causal questions, experimental approaches are preferred. When experimental approaches are not feasible, high-quality quasi-experiments offer an alternative.” ACL is in the process of implementing a method for rating each proposed evaluation against OMB’s Program Evaluation Standards and Practices as defined in OMB M-20-12.

Did the agency invest at least 1% of program funds in evaluations in FY20? (Examples: Impact studies; implementation studies; rapid cycle evaluations; evaluation technical assistance, rigorous evaluations, including random assignments)

3.1 ____ (Name of agency) invested $____ on evaluations, evaluation technical assistance, and evaluation capacity-building, representing __% of the agency’s $___ billion FY20 budget.

• ACL invested $22.2 million on evaluations, evaluation technical assistance, and evaluation capacity-building, representing 1.11% of the agency’s $2.0 billion FY20 enacted budget.

3.2 Did the agency have a budget for evaluation and how much was it? (Were there any changes in this budget from the previous fiscal year?)

• ACL’s Office of Performance and Evaluation (OPE) budget for evaluation was $10.4 million in FY20 and there were no significant changes to the evaluation budget since the previous year. The bulk of OPE’s evaluation funds are based on a set-aside required in Title II, section 206, of the Older Americans Act, “From the total amount appropriated for each fiscal year to carry out title III, the Secretary may use such sums as may be necessary, but not to exceed 1⁄2 of 1 percent of such amount, for purposes of conducting evaluations under this section, either directly or through grants or contracts.” In addition, in 2017 ACL’s Office of Performance and Evaluation established a mechanism that allows ACL programs not covered by the OAA set-aside to transfer funds to OPE to be able to support evaluations of their programs. In 2017, 2018, 2019, and 2020 OPE added approximately $1.0 million, $1.7 million, $3.2 million, and $1.2 million from these programs to its evaluation budget respectively. In addition to these funds, ACL allocated $11.8 million for technical assistance and other activities, such as prize competitions, to support strong evidence building. In total, ACL spent $22.2 million on evaluation related activities in FY20.

3.3 Did the agency provide financial and other resources to help city, county, and state governments or other grantees build their evaluation capacity (including technical assistance funds for data and evidence capacity building)?

• ACL primarily provides information resources to grantees to build their evaluation and evidence building capacity. Staff record trainings on evaluation topics, including an overview of performance measurement. ACL also has several resources and TA centers that focus on evidence building including one contract dedicated to improving performance data provided by Older Americans Act Title III, VI, and VII grantees that offers live and prerecorded webinars and a range of manuals and TA supports. ACL also published toolkits for strategic planning, data quality, performance measures, and logic model development. ACL provides technical assistance to grantees related to using evidence-based programs and building evidence. For example, the National Resource Center on Nutrition and Aging (NRC) provides different programs and approaches that deliver nutrition-related home- and community-based services (HCBS) administered through grants to the 56 states and territories. ACL’s Alzheimer’s Disease Supportive Services Program (ADSSP) grant program supports state efforts to expand the availability of community-level supportive services including the translation of evidence-based models into community-level practice in their programs. The 68 University Centers for Excellence in Developmental Disabilities Education, Research, and Service (UCEDDs) throughout the United States and its territories serve as liaisons between academia and the community. They fund model demonstrations to build evidence for addressing issues, finding solutions, and advancing research related to the needs of individuals with developmental disabilities and their families.
• Access to Respite Care and Help (ARCH) provides training and technical assistance to the Lifespan Respite Network with a focus on performance measurement, sustainability, best practices, and research. ACL has funded 33 states and the District of Columbia to establish or enhance Statewide Lifespan Respite systems and ARCH provides training and technical assistance to them as well. The National Alzheimer’s and Dementia Resource Center supports grantees as they implement evidence-based interventions and innovative practices designed to empower and assist caregivers of persons with Alzheimer’s disease and related disorders. ACL funding is released in five year cycles, with the next five year grant expected to be awarded in 2020.

Did the agency implement a performance management system with outcome-focused goals and aligned program objectives and measures, and did it frequently collect, analyze, and use data and evidence to improve outcomes, return on investment, and other dimensions of performance in FY20?
(Example: Performance stat systems, frequent outcomes-focused data-informed meetings)

4.1 Did the agency have a strategic plan with outcome goals, program objectives (if different), outcome measures, and program measures (if different)?

• As part of the U.S. Department of Health and Human Services Annual Performance Plan and Report, ACL reports on the following two HHS Agency Priority Goals: (1) Increase the success rate of the Protection and Advocacy Program’s individual or systemic advocacy, thereby advancing individuals with developmental disabilities’ right to receive appropriate community based services, resulting in community integration and independence, and have other rights enforced, retained, restored and/or expanded; and (2) Improve dementia capability of long-term support systems to create dementia-friendly, livable communities (Lead Agency ACL). ACL’s strategy focuses on five pillars: supporting families and caregivers, protecting rights and preventing abuse, connecting people to resources, expanding employment opportunities, and strengthening the aging and disability networks. These pillars provide structure and focus for ACL’s work. ACL’s outcomes measures are available, by program, in its annual Congressional Budget Justification, and include measures of program efficiency.

4.2 Does the agency use data/evidence to improve outcomes and return on investment?

• ACL employs a moderate approach for analyzing evidence to find ways to improve return on investment that addresses multiple parts of the agency. In FY20, as part of its ongoing effort to ensure that agency funds are used effectively, ACL funded a contract, focused on ACL’s Administration in Aging, to identify approaches to measure how and to what extent parts of the Aging Network leverage Older Americans Act funds to increase their available resources as well as how the Aging Network uses resources to measure and improve the quality of services available/provided. NIDILRR conducts research as part of their new employment research agenda to continue development of return-on-investment models that can be used by Vocational Rehabilitation agencies to optimize the services they provide. In addition, in March 2020 ACL launched a Challenge Competition to spur development of the interoperable, statewide referral and analytics platforms needed to enable the type of partnerships between health care and community-based social services organizations have been shown to improve health outcomes and lower costs.
• In June 2020 ACL launched MENTAL Health Challenge to create an online tool that connects socially isolated people to resources. In November 2020, ACL launched two competitions. The Inventive Solutions to Address the Direct Support Professional Crisis has the goal of improving the overall quality of home- and community-based services (HCBS) for individuals with intellectual and developmental disabilities (ID/DD). The Disability Employment Challengesought innovative models that can be shared to help businesses across the country reach a wider talent pool and to create more opportunities for employment for people with disabilities. The goal of all the prize competitions is to encourage effective and efficient methods for meeting ACL’s mission and improving services to its target populations.

4.3 Did the agency have a continuous improvement or learning cycle processes to identify promising practices, problem areas, possible causal factors, and opportunities for improvement? (Examples: stat meetings, data analytics, data visualization tools, or other tools that improve performance)

• As part of ACL’s performance strategy and learning agenda approach OPE staff present performance data to ACL leadership several times a year. In addition, ACL leadership review performance data as part of the budget justification process that informs program funding decisions. OPE staff conduct annual meetings with ACL staff to report performance measure data and results to discuss methods for incorporating performance and evaluation findings into funding and operational decision-making. As part of annual evaluation planning efforts, staff from ACL’s Office of Performance and Evaluation consult with ACL center directors to identify evaluation priorities and review proposed evaluation approaches to ensure that the evaluation questions identified will provide information that will be useful for program improvement.

Did the agency collect, analyze, share, and use high-quality administrative and survey data – consistent with strong privacy protections – to improve (or help other entities improve) outcomes, cost-effectiveness, and/or the performance of federal, state, local, and other service providers programs in FY20? (Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; open data policies; data-use policies)

5.1 Did the agency have a strategic data plan, including an open data policy? (Example: Evidence Act 202(c), Strategic Information Resources Plan)

• As an operating division of a CFO Act Agency, the U.S. Department of Health and Human Services, ACL is not required to have its own strategic data plan and utilizes HHS’s data strategy. In 2016, ACL implemented a Public Access Plan as a mechanism for compliance with the White House Office of Science and Technology Policy’s public access policy. The plan focused on making published results of ACL/National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) funded research more readily accessible to the public; making scientific data collected through ACL/NIDILRR-funded research more readily accessible to the public; and increasing the use of research results and scientific data to further advance scientific endeavors and other tangible applications. In 2019, ACL created a council to improve ACL’s data governance, including the development of improved processes and standards for defining, collecting, reviewing, certifying, analyzing, and presenting data that ACL collects through its evaluation, grant reporting, and administrative performance measures. In 2020, its first year, the ACL Data Council produced an annotated bibliography to provide essential background information about the topic, and also developed a Primer to detail best practices in data governance specifically as they apply to ACL. The Council also developed a Data Quality 101 infographic to guide decision-making processes related to data quality, which will be completed in late 2020.

5.2 Did the agency have an updated comprehensive data inventory? (Example: Evidence Act 3511)

• ACL provides comprehensive public access to its programmatic data through two web based portals: Aging, Independence, and Disability (AGID) and National Rehabilitation Information Center (NARIC). ACL is also creating an internal evidence inventory that staff will be able to use to search for relevant program performance and evaluation data by agency priority question.

5.3 Did the agency promote data access or data linkage for evaluation, evidence-building, or program improvement? (Examples: Model data-sharing agreements or data-licensing agreements; data tagging and documentation; data standardization; downloadable machine-readable, de-identified tagged data; Evidence Act 3520(c)) 

• ACL’s Office of Performance and Evaluation has access to all of ACL’s performance and evaluation data and is able to link those data and advise programs about their availability and usability. In March 2019, the ACL completed the ACL Data Restructuring (DR) Project to assess the data hosted on AGID, and to develop and test a potential restructuring of the data in order to make it useful and usable for stakeholders. In 2019, ACL awarded a follow on contract to further integrate its datasets along the lines of conceptual linkages, and to better align the measures within ACL’s data collections across the agency. ACL funded several grants to promote data linkage including the Grants to Enhance State Adult Protective Services awarded in FY19 to increase intra- and inter-state sharing of information on APS cases and the 2020 Empowering Communities to Reduce Falls and Falls Risk to develop robust partnerships, develop a result-based, comprehensive strategy for reducing falls and falls risk among older adults and adults with disabilities living in your community and directs grantees to consider CDC opportunities to broaden and improve the linkage between primary care providers and evidence-based community falls prevention programs supported by ACL.

5.4 Did the agency have policies and procedures to secure data and protect personal, confidential information? (Example: differential privacy; secure, multiparty computation; homomorphic encryption; or developing audit trails)\

• As an operating division of the U.S. Department of Health and Human Services, ACL follows all departmental guidance regarding data privacy and security. This includes project-specific reviews by ACL’s Office of Information Resource Management (OIRM), which monitors all of ACL’s data collection activities to ensure the safety and security of ACL’s data assets. In FY19, ACL awarded a contract to stand up a “Data Council” to enhance the quality, security, and statistical usability of the data ACL collects through its evaluation, grant reporting, and administrative data collections, and to develop effective data governance standards. A Data Governance and a Data Quality working group have been established to actively address these goals. In addition, each funding opportunity announcement states that “a data and safety monitoring board (DSMB) is required for all multi-site clinical trials involving interventions” (see for example the FOA for Disability and Rehabilitation Research Projects (DRRP): Assistive Technology to Promote Independence and Community Living (Development) HHS-2019-ACL-NIDILRR-DPGE-0355).

5.5 Did the agency provide assistance to city, county, and/or state governments, and/or other grantees on accessing the agency’s datasets while protecting privacy?

• ACL staff provide technical assistance through presentations and ACL’s technical assistance resource centers to grantees, including state, tribal, and local governments. The resource centers providing technical assistance include: the National Resource Center on Nutrition and Aging (NRC), the Alzheimer’s Disease Supportive Services Program(ADSSP) and the University Centers for Excellence in Developmental Disabilities Education, Research, and Service. This technical assistance includes annual workshops and presentations at the Title VI National Training and Technical Assistance Conference; training available through the ACL funded National Ombudsman Resource Center; and the Disability and Rehabilitation Research Program (DRRP), which funds capacity building for minority research entities.

Did the agency use a common evidence framework, guidelines, or standards to inform its research and funding purposes; did that framework prioritize rigorous research and evaluation methods; and did the agency disseminate and promote the use of evidence-based interventions through a user-friendly tool in FY200? (Example: What Works Clearinghouses)

6.1 Did the agency have a common evidence framework for research and evaluation purposes?

• ACL defines evidence-based programs on its website. ACL’s National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) uses a stages of research framework (SORF) to classify and describe its funded grants and research projects within the grants. Rigorous evaluations methods are applied based on appropriateness. The four stages of SORF include: exploration and discovery, intervention development, intervention efficacy, and scale-up evaluation. Using SORF, NIDILRR gains insight into what is known and unknown about a problem; whether it is time to develop interventions to address a particular problem; whether it is time to test the efficacy of interventions; and whether it is time to scale-up interventions for broader use.

6.2 Did the agency have a common evidence framework for funding decisions?

• The Older Americans Act requires the use of evidence-based programming in Title III-D-funded activities: Disease Prevention and Health Promotion Services. In response, ACL developed a definition of the term evidence-based, and created a website containing links to a range of resources for evidence-based programs. This is a common evidence framework used for Older Americans Act funded activities. For programs that are not legislatively required to use evidence-based models, through its funding process ACL requires all programs to provide clear justification and evidence (where available) that proposed projects will achieve their stated outcomes.

6.3 Did the agency have a user friendly tool that disseminated information on rigorously evaluated, evidence-based solutions (programs, interventions, practices, etc.) including information on what works where, for whom, and under what conditions?

• ACL does not have a common evidence repository that applies across the entire agency. It publishes intervention summaries of aging and disability evidence-based programs and practices. ACL funds the Evidence-Based Program Review Council to identify new community programs that meet the criteria established by the Administration for Community Living/Administration on Aging (ACL/AoA) for evidence-based programs funded through the Older Americans Act (OAA) Title III-D. To support the use of evidence-based and evidence-informed programming, service providers can find out about evidence-based programs that serve people with dementia and their caregivers by consulting a white paper drafted with funds from ACL– Translating Innovation to Impact: Evidence-based interventions to support people with Alzheimer’s disease and their caregivers at home and in their communities. The Model Systems Knowledge Translation Center (MSKTC) has worked with NIDILRR’s Model Systems grantees to develop and publish a variety of evidence-based factsheets about living with spinal cord injury, traumatic brain injury, or burn injury. ACL’s Living Well demonstration program requires grantees to useevidence-based and innovative strategies to (1) improve access to and quality of community services, (2) reduce and mitigate abuse and neglect, and (3) support empowerment, self-determination, and self-advocacy.
  

6.4 Did the agency promote the utilization of evidence-based practices in the field to encourage implementation, replication, and application of evaluation findings and other evidence?

• ACL works through its resource centers to help grantees use evidence to drive improvements in outcomes for older adults and individuals with disabilities. For example, with funding from ACL, the National Resource Centersat National Center on Aging (NCOA), in collaboration with the Evidence-Based Leadership Council, led an innovative vetting process to increase the number of programs available to ACL’s aging network that meet the Title III-D evidence-based criteria. This process resulted in adding six new health promotion programs and three new programs for preventing falls. The Alzheimer’s Disease Supportive Services Program (ADSSP) funds competitive grants to expand the availability of evidence-based services that support persons with Alzheimer’s disease and related dementia (ADRD) and their family caregivers. Extensive evaluation of the National Chronic Disease Self-Management Education (CDSME) and Falls Prevention database helped generate important insights for potential new ACL applicants in preparing their applications using data-driven estimation procedures for participant and completion targets (see Guidance for Administration for Community Living 2019 Chronic Disease Self-Management Education Grant Applicants: Considerations for Estimating Participation and Completion Targets). ACL also funded several grants, such as the Lifespan Respite Care Program: State Program Enhancement Grantsand Disability and Rehabilitation Research Projects (DRRP) Program: Chronic Disease Management for People with Traumatic Brain Injury (TBI) which are designed, in part, to develop an evidence base for respite care and related services and contribute to the evidence base upon which people with TBI and their health care providers can employ effective chronic disease management practices respectively .
• Starting in FY20, ACL is also conducting an evaluation of the fidelity with which ACL and its grantees under the Older Americans Act are implementing the required evidence-based programs. This will result in a report documenting the information collected and providing clear, actionable recommendations for ensuring the effective use of evidence-based programming. Recommendations will address what ACL, its grantees, and sub-grantees can do to improve the selection, implementation, and monitoring of evidence-based programming. The report will also include the development of a standardized tool for use by ACL and its OAA state grantees that assesses evidence-based program fidelity. This tool will greatly enhance ACL’s ability to ensure that evidence-based practices are used in the field.

Did the agency have staff, policies, and processes in place that encouraged innovation to improve the impact of its programs in FY20?
(Examples: Prizes and challenges; behavioral science trials; innovation labs/accelerators; performance partnership pilots; demonstration projects or waivers with rigorous evaluation requirements)

7.1 Did the agency engage leadership and staff in its innovation efforts to improve the impact of its programs?

• Agency leadership promotes innovation by requiring all program offices to explain, in their annual funding proposals, how the proposed use of funds will identify innovative practices. ACL also partially funds a Forum on Aging, Disability, and Independence which engages staff to foster discussions about innovation for coordinating and integrating aging and disability stakeholders. The most recent publication from this group, released in 2019 is Artificial Intelligence Applications for Older Adults and People with Disabilities: Balancing Safety and Autonomy: Proceedings of a Workshop—in Brief. ACL also funds resource centers, such as the Engagement and Older Adults Resource Center which provides technical assistance and serves as a repository for innovations designed to increase the aging network’s ability to tailor social engagement activities to meet the needs of older adults.
  

7.2 Did the agency have policies, processes, structures, or programs to promote innovation to improve the impact of its programs?

• In FY20 all ACL Centers were involved in funding innovative work. ACL released several funding opportunity announcements (FOA) focused on the identification and implementation of innovative approaches to improve programming. These included the Innovations in Nutrition Programs and Services – Demonstration , the Small Business Innovation Research (SBIR) Program, and Grants to Enhance State Adult Protective Servicesdemonstration addresses gaps and challenges in state APS systems through innovations and improvements in practice, services, data collection, and reporting. ACL also funded challenge competitions to spur development of the interoperable, statewide referral and analytics platforms needed to enable these partnerships ( IT Solutions) and to combat the social isolation and loneliness that older adults, people with disabilities and veterans often experience (Mental Health Challenge).
• ACL is a funder of The John A. Hartford Foundation 2020 Business Innovation Award which recognizes aging and disability community-based organizations (CBOs) for their innovative approaches to reducing health care costs and improving the well-being of older adults and people with disabilities through strategic partnership with health care entities.
• There are several funding streams that support innovation. The Older Americans Act, which funds ACL’s Administration on Aging, allows ACL to use up to 1% of its appropriations for nutrition innovation demonstrations designed to develop and implement evidence-based practices that enhance senior nutrition. One result is that, consistent with the Administrator’s focus on identifying new ways to efficiently improve direct service programs, ACL is using $3.5 million to fund nutrition innovations and test ways to modernize how meals are provided to a changing senior population. One promising demonstration (entitled Double Blind Randomized Control Trial on the Effect of Evidence-Based Suicide Intervention Training on the Home-Delivered and Congregate Nutrition Program through the Atlanta Regional Commission), currently being carried out by the Georgia State University Research Foundation, is an effort to train volunteers who deliver home-delivered meals to recognize and report indicators of suicidal intent and other mental health issues so that they can be addressed.
• State Councils on Developmental Disabilities (SCDD) are charged with identifying and addressing the most pressing needs of people with developmental disabilities in their state and territory. Councils work with different groups in many ways, including educating communities to welcome people with developmental disabilities; funding projects to show new ways that people with disabilities can work, play, and learn; and seeking information from the public as well as state and national sources.
• The 2020 reauthorization of the Older Americans Act has provisions for technical assistance and innovation to improve transportation for older individuals.
• In 2020, ACL awarded grants for demonstrations in Innovations in Nutrition Programs and Services to support the documentation of innovative projects that enhance the quality, effectiveness, and other proven outcomes of nutrition services programs within the aging services network. The goal of this funding opportunity is to support projects that can demonstrate potential for broad implementation throughout the aging services network, and with demonstrated value, i.e., improvements in participant well-being, cost savings, etc.
• As previously mentioned, National Institute on Disability, Independent Living, and Rehabilitation Research(NIDILRR) uses a stages of research framework (SORF) process to gain insight into what is known and unknown about a problem; whether it is time to develop interventions to address a particular problem; whether it is time to test the efficacy of interventions (evaluate its innovation efforts); and whether it is time to scale-up interventions for broader use (improve the impact of its programs).

7.3 Did the agency evaluate its innovation efforts, including using rigorous methods?

• The 2020 reauthorization of the Older Americans Act requires a new Research, Demonstration, and Evaluation Center for the Aging Network and new demonstration programs to evaluate new strategies for the recruitment, retention, or advancement of direct care workers, and the soliciting, development, and implementation of strategies; and a demonstration to address negative health impacts associated with social isolation. Further, ACL has a number of model programs and demonstration grants that propose and test the use of innovative approaches. For example, ACL funded cooperative agreements for the development and testing of model approaches towards coordinated and comprehensive systems for enhancing and assuring the independence, integration, safety, health, and well-being of individuals with intellectual and developmental disabilities living in the community (i.e. Living Well Grants). While the evaluation of this program is not yet complete, initial findings about what works were integrated into the requirements of the funding announcement for the FY18 award cycle.
• As previously mentioned, NIDILRR’s research and development activities are guided by the Stages of Research Framework and the Stages of Development Framework. NIDILRR grantees conducting research projects must identify the stage of research their projects are in while grantees conducting development projects must identify the stage of development their projects are in. The stage a research project is in depends upon what is known and what is not known about a particular disability problem or topic. Research projects where relatively little is known, or the topic area is emerging, would be classified in the Exploration and discovery stage. Over time, as more becomes known, research projects progressively move into the Intervention Development phase. The next phase, known as Intervention Efficacy, means the stage of research during which a project evaluates and tests whether an intervention is feasible, practical, and has the potential to yield positive outcomes for individuals with disabilities. The final stage, known as Scale-Up Evaluation, means the stage of research during which a project analyzes whether an intervention is effective in producing improved outcomes for individuals with disabilities when implemented in a real-world setting.
• Similarly, the stage of development a development project is in also depends upon what is known or not known about a need that informs the design and development of a product. The proof of concept stage means the stage of development where key technical challenges are resolved. Stage activities may include recruiting study participants, verifying product requirements; implementing and testing (typically in controlled contexts) key concepts, components, or systems, and resolving technical challenges. The proof of product stage means the stage of development where a fully-integrated and working prototype, meeting critical technical requirements is created. The proof of adoption stage means the stage of development where a product is substantially adopted by its target population and used for its intended purpose.

Did the agency use evidence of effectiveness when allocating funds from its competitive grant programs in FY20? (Examples: Tiered-evidence frameworks; evidence-based funding set-asides; priority preference points or other preference scoring for evidence; Pay for Success provisions)

8.1 What were the agency’s five largest competitive programs and their appropriations amount (and were city, county, and/or state governments eligible to receive funds from these programs)?

• In FY20, the five largest competitive grant programs are:
  1. National Institute on Disability, Independent Living, and Rehabilitation Research (NIDILRR) ($112.0 million; eligible applicants: State, local, and tribal governments and nonprofits, public and State controlled institutions of higher education)
     • NIDILRR’s largest competitive grants are its Disability and Rehabilitation Research Projects (DRRP)
  2. Centers for Independent Living ($90.8 million; eligible applicants: Nonprofits; Public and State controlled institutions of higher education)
     • One of their largest competitive grants for was the Centers for Independent Living Training and Technical Assistance Grant
  3.  State Health Insurance Assistance Program ($52.1 million; eligible applicants: Unrestricted)
     • One of the relevant NOFAs is for 2020 State Health Insurance Assistance Program (SHIP) Base Grant
  4. University Centers for Excellence in Developmental Disabilities Education, Research and Service ($41.6 million; eligible applicants: entities in each State designated as UCEDDs to carry out the four core functions of interdisciplinary pre-service preparation and continuing education, community services, research, and information dissemination)
  5. Medicare Improvements for Patients and Providers Act Programs (MIPPA) ($38 million; Eligible applicants are: Nonprofits; City or township governments; Public and State controlled institutions of higher education; Native American tribal; Public housing authorities/Indian housing authorities; Private institutions of higher education; Native American tribal organizations; Special district governments; County governments; State governments; and Independent school districts).
     • A relevant NOFA is for the National Center for Benefits Outreach & Enrollment

8.2 Did the agency use evidence of effectiveness to allocate funds in five largest competitive grant programs? (e.g., Were evidence-based interventions/practices required or suggested? Was evidence a significant requirement?)

• Based on a strict interpretation of the phrase “evidence of prior effectiveness to make grant awards,” NIDILRR currently does not use evidence of prior effectiveness to make grant awards. Instead, ACL makes these grant awards by largely relying on the expert evaluative judgments of our peer reviewers. Making grant awards by using peer review is a standard, and widely-accepted, evidenced-based practice. For example, see page 7 of the full DPCP full announcement.
• Independent Living (IL) NOFAs describe evaluation criteria including plans for technical assistance to enhance grant effectiveness and the provision of information developed about best practices (full announcement (p. 21)). To continue receiving CIL program funding, eligible centers must provide evidence that they have previously had an impact on the goals and objectives for this funding.
• SHIP NOFAs describe evaluation criteria including plans to improve alignment of policies, processes, and procedures to program goals and increased accountability to program expectations at all levels (full announcement (p.25)).
• University Centers for Excellence in Developmental Disabilities Education, Research & Service (UCEDDs) are a nationwide network of independent but interlinked centers, representing an expansive national resource for addressing issues, finding solutions, and advancing research related to the needs of individuals with developmental disabilities and their families. According to the funding opportunity announcement applications are also reviewed based on their description of current or previous evidence of relevant experience.
• MIPPA funds are awarded to State grantees and to the National Center for Benefits Outreach and Enrollment. To continue funding without restrictions, State grantees are required to submit state plans that ACL staff review for the specific strategies that grantees will employ to enhance efforts through statewide and local coalition building. TheNational Center applicants must describe the rationale for using the particular intervention, including factors such as evidence of intervention effectiveness. In 2019, the Center was awarded additional funding based on prior performance— specifically, assisting over 7.6 million individuals to identify over $29.6 billion in potential annual benefits.

8.3 Did the agency use its 5 largest competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations)

• NIDILRR, and its grantees, are in the disability and rehabilitation evidence-building business. NIDILRR grantees generate new knowledge, on particular disability topics or develop new disability products which eventually becomes part of a larger evidence base. To generate this new knowledge, NIDILRR grantees must conduct a series of research and development activities that produce important outputs. These research and development activities are guided by the following two frameworks: The NIDILRR Stages of Research Framework, and the NIDILRR Stages of Development Framework. The NIDILRR Stages of Research Framework is published in 45 CFR 1330.4 while the Stages of Development Framework is published in 45 CFR 1330.5.
• Independence Living/Centers for Independent living grants are required to show that they are “improving performance, outcomes, operations, and governance of CILs.” (Full Announcement (p. 5))
• SHIP grantees are required to build and disseminate evidence of what works through documenting and promoting “knowledge, successes, and lessons learned within the SHIP network. This includes sharing ideas, products, and materials with other SHIP grantees, ACL, and the SHIP Technical Assistance Center.” (Full Announcement (p.5)).
• A central purpose of UCEDD grants is the building and dissemination of evidence of what works. UCEDDs are a nationwide network of independent but interlinked centers, representing an expansive national resource for addressing issues, finding solutions, and advancing research related to the needs of individuals with developmental disabilities and their families.
• MIPPA Grant funds support the identification and dissemination of promising practices for (i.e., practices built upon evidence of effectiveness) improving benefits outreach and enrollment.

8.4 Did the agency use evidence of effectiveness to allocate funds in any competitive grant program?

• ACL requires that evidence of effectiveness is used in all award decisions. Grant officers attend training regarding ways to include information about evidence building into funding opportunity announcements. This includes information about text that can be included in funding announcements: 1) describing requirements for developing measurable outcomes; 2) explaining how the inclusion of evidence and evidence building plans can be used to score grant applications; and 3) instructing grant reviewers regarding rating applicants’ presentation of evidence and evidence building plans. The training was recorded and is available to all staff.
• ACL’s Alzheimer’s Disease Programs Initiative (ADPI) translates and implements evidence-based supportive services for persons with ADRD and their caregivers at the community level. Award criteria include the extent to which applicants ” describe partnerships, collaborations and innovative activities that will be implemented in support of goal/objective achievement, including the dementia specific evidence-based/evidence informed intervention(s) to be implemented in the project” (Full Announcement (p. 24))
• The review criteria for the Lifespan Respite Care Program: State Program Enhancement Grants includes the applicant’s description of “how the proposed project will build upon the accomplishments made in previous Lifespan Respite Care Program grants” (Full Announcement (p. 23)).

8.5 What are the agency’s 1-2 strongest examples of how competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?

• Prior to the development of visual scene displays by the NIDILRR-funded Augmentative and Alternative Communication Rehabilitation Engineering Research Center (AAC-RERC), the only Augmentative and Alternative Communication (AAC) option was traditional grid displays with isolated symbols presented in rows and columns. It was difficult for many adults with acquired conditions resulting in significant language and cognitive limitations to use these traditional grid displays. Visual Scene Displays (VSDs) offer an easier alternative to traditional grid displays. They go beyond standard pictures and symbols organized in rows and columns by providing information on the situation or context. Put more simply, VSDs are photos or pictures that people can use to communicate messages to others. These photos depict familiar scenes, objects or people—and users can touch “hot spots” on the photo to speak messages that relate to the pictured scene or object. For example, a person with aphasia might touch a hotspot on a picture of a sibling and say this is my sister. This additional information on the situation and context makes it easier for persons with complex communication needs to express their wants and needs and therefore enhances their ability to interact and participate with others in the community. Research from the AAC RERC and external researchers demonstrates the effectiveness of VSDs with adults with severe chronic aphasia, primary progressive aphasia, dementia, etc. As a result of the continued efforts of the AAC-RERC and their partners, this VSD technology has been successfully transferred to all of the major AAC manufacturers and app developers.
• ACL’s Alzheimer’s Disease Supportive Services Program (ADSSP) encourages the translation of dementia specific interventions for use in communities. Examples include: the Savvy Caregiver (evidence-based) psychoeducational intervention focused on training family caregivers about the basic knowledge, skills, and attitudes needed to handle the challenges of caring for a family member with Alzheimer’s disease and to be an effective caregiver; Cuidando con Respeto (evidence-informed), Spanish version of the original Savvy Caregiver Program; and Savvy Caregiver Express (evidence-informed), a condensed version of the original Savvy Caregiver Program. ACL’s requirement for inclusion of dementia specific evidence-based interventions is demonstrated in the 2018 funding opportunity announcement entitled Alzheimer’s Disease Programs to States and Communities.

8.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?

• Funding opportunity announcements and grant reviews stress the need for strong performance measurement and evaluation. ACL’s technical assistance centers— the National Resource Center on Nutrition and Aging (NRC), the Alzheimer’s Disease Supportive Services Program (ADSSP) and the University Centers for Excellence in Developmental Disabilities Education, Research, and Service— promote the use and generation of evidence with ACL grantees. Grantees manuals also include information about the importance of and requirements for evaluation (see the Administration on Aging: Title VI Resource Manual). Staff of ACL’s Office of Performance and Evaluation make presentations regarding the importance of evidence with regional staff who are in frequent contact with State grantees and at grantee conferences (see ACL Track: The ACL Older Americans Act (OAA) Performance System – Crossing the Finish Line and ACL/CMS Track: Raising the Bar in Medicaid HCBS & Community Inclusion – Showcasing Transformation presented at the 2019 home- and community-based services (HCBS) conference; ACL Track: Assuring the Health & Welfare of Medicaid HCBS Beneficiaries: Federal Findings, Investments, & Promising Practices in Systems Change and ACL Track: Innovative Housing & Health & Human Services Collaborations: A Game-Changer in Supportive Housing & Community Living presented at the 2018 HCBS conference).

Did the agency use evidence of effectiveness when allocating funds from its non-competitive grant programs in FY20?
(Examples: Evidence-based funding set-asides; requirements to invest funds in evidence-based activities; Pay for Success provisions)

9.1 What were the agency’s five largest non-competitive programs and their appropriation amounts (and were city, county, and/or state governments are eligible to receive funds from these programs)?

• In FY20, the five largest non-competitive grant programs are:
  1. Nutrition Services, ($936.8 million; eligible applicants: States)
  2. Home and Community Based Supportive Services ($390.1 million; eligible applicants: States);
  3. Caregiver Support Services ($185.9 million; eligible applicants: States)
  4. State Councils on Developmental Disabilities ($78 million; eligible applicants: States and nonprofits based in a State);
  5. Developmental Disabilities Protection and Advocacy ($40.1 million; eligible applicants: State and nonprofits based in a States)
• As these are based on formula grants, the funding amount distributed to the States and tribal organizations are not determined using evidence-based application processes. Rather, the States and tribal organizations are responsible for directing the funds to evidence-based programs and organizations.
  

9.2 Did the agency use evidence of effectiveness to allocate funds in its largest five non-competitive grant programs? (e.g., Are evidence-based interventions/practices required or suggested? Is evidence a significant requirement?)

• Authorizing legislation for ACL’s largest non-competitive grant programs requires consideration of evidence-based programming as a requirement of funding. The Developmental Disabilities Assistance and Bill of Rights Act of 2000 allows for the withholding of funding if (1) the Council or agency has failed to comply substantially with any of the provisions required by section 124 to be included in the State plan, particularly provisions required by paragraphs (4)(A) and (5)(B)(vii) of section 124(c), or with any of the provisions required by section 125(b)(3); or (2) the Council or agency has failed to comply substantially with any regulations of the Secretary that are applicable.” As a condition of funding non-competitive grandees are required to “determine the extent to which each goal of the Council was achieved for that year” and report that information to ACL.
• States that receive Older Americans Act Home and Community-Based Supportive Services Title III-D funds are required to spend those funds on evidence-based programs to improve health and well-being, and reduce disease and injury. In order to receive funding, states must utilize programs that meet ACL’s definition of evidence-based or are defined as evidence-based by another HHS operating division. Under the Older American Act, caregiver support programs are required to track and report on their use of evidence-based caregiver support services.

9.3 Did the agency use its five largest non-competitive grant programs to build evidence? (e.g., requiring grantees to participate in evaluations)

• FY12 Congressional appropriations included an evidence-based requirement for the first time. OAA Title III-D funding may be used only for programs and activities demonstrated to be evidence-based. Consistent with the Administrator’s focus on identifying new ways to efficiently improve direct service programs, ACL is using its 1% Nutrition authority to fund $3.5 million for nutrition innovations and to test ways to modernize how meals are provided to a changing senior population. One promising demonstration currently being carried out by the Georgia State University Research Foundation (entitled Double Blind Randomized Control Trial on the Effect of Evidence-Based Suicide Intervention Training on the Home-Delivered and Congregate Nutrition Program through the Atlanta Regional Commission) which has drawn widespread attention is an effort to train volunteers who deliver home-delivered meals to recognize and report indicators of suicidal intent and other mental health issues so that they can be addressed.

9.4 Did the agency use evidence of effectiveness to allocate funds in any other non-competitive grant programs in FY20 (besides its five largest grant programs)?

• The 2020 reauthorization of the Older Americans Act requires that Assistive technology programs are “aligned with evidence-based practice;” that person-centered, trauma informed programs “incorporate evidence-based practices based on knowledge about the role of trauma in trauma victims’ lives;” and that a newly authorized Research, Demonstration, and Evaluation Center for the Aging Network increases “the repository of information on evidence based programs and interventions available to the aging network, which information shall be applicable to existing programs and interventions, and help in the development of new evidence-based programs and interventions.”

9.5 What are the agency’s 1-2 strongest examples of how non-competitive grant recipients achieved better outcomes and/or built knowledge of what works or what does not?

• Since 2017, ACL has awarded Innovations in Nutrition grants to 11 organizations to develop and expand evidence-based approaches to enhance the quality and effectiveness of nutrition programming. ACL is currently overseeing five grantees for innovative projects that will enhance the quality, effectiveness, and outcomes of nutrition services programs provided by the national aging services network. The grants total $1,197,205 for this year with a two-year project period. Through this grant program, ACL aims to identify innovative and promising practices that can be scaled across the country and to increase the use of evidence-informed practices within nutrition programs.
• In 2020, ACL expects to award grants for demonstrations in Innovations in Nutrition Programs and Services to support the documentation of innovative projects that enhance the quality, effectiveness, and other proven outcomes of nutrition services programs within the aging services network.

9.6 Did the agency provide guidance which makes clear that city, county, and state government, and/or other grantees can or should use the funds they receive from these programs to conduct program evaluations and/or to strengthen their evaluation capacity-building efforts?

•  All funding opportunity announcements published by ACL include language about generating and reporting evidence about their progress towards the specific goals set for the funds. Grantee manuals include information about the importance of and requirements for evaluation (see the Administration on Aging: Title VI Resource Manual). The National Ombudsman Resource Center, funded by ACL, provides self-evaluation materials for Long-Term Care Ombudsman Programs (LTCOP) funded under Title VII of the Older Americans Act.

In FY20, did the agency shift funds away from or within any practice, policy, or program that consistently failed to achieve desired outcomes?
(Examples: Requiring low-performing grantees to re-compete for funding; removing ineffective interventions from allowable use of grant funds; incentivizing or urging grant applicants to stop using ineffective practices in funding announcements; proposing the elimination of ineffective programs through annual budget requests; incentivizing well-designed trials to fill specific knowledge gaps; supporting low-performing grantees through mentoring, improvement plans, and other forms of assistance; using rigorous evaluation results to shift funds away from a program)

10.1 Did the agency have a policy for determining when to shift funds away from grantees, practices, policies, interventions, and/or programs that consistently failed to achieve desired outcomes, and did the agency act on that policy?

• Because much of ACL’s funding is based on formula grants that cannot be reallocated to other programs or grantees, there is not an ACL-wide policy for this purpose. For several programs, such as most under the Older American Act, “entities such as states, U.S. territories, and tribal organizations are allotted funding based on a population-based formula factor (e.g., aged 55 and over, aged 60 and over, or aged 70 and over). Some statutory requirements for program funding allocations include a “hold harmless” provision, which guarantees that state or other entities’ allotment will remain at a certain fiscal year level or amount, provided sufficient funding in a given year. ACL is working with GSA’s Office of Evaluation Sciences (OES) to test methods for improving outcomes for its congregate meals programs. Under the Older Americans Act, congregate meal sites are required to accept donations from meal recipients. But, there has been a concern regarding how to balance the collection of funds that can be used towards meal service and making meal recipients that cannot afford to donate uncomfortable, thus suppressing attendance. This study, expected to be completed in FY 2020, will offer concrete evidence to improve program operations.

10.2 Did the agency identify and provide support to agency programs or grantees that failed to achieve desired outcomes?

• While much of ACL’s funding is based on formula grants, and therefore cannot be reallocated to other programs, evaluation staff work closely with program staff to identify ways to translate evaluation findings into technical assistance and other types of program support. For example, based on early results from an evaluation of the Tribal Grant program, ACL has developed new program support materials to improve the delivery of Tribal Caregiver programs.
• ACL typically proactively provides technical assistance in order to help programs to be successful, rather than redirecting funding. For example, the State Health Insurance Assistance Program (SHIP) is funding a national Technical Assistance center for this purpose.