How NIH Is Using Artificial Intelligence To Improve Operations

Artificial intelligence (AI) is everywhere, from the online marketplace to the laboratory! When you read an article or shop online, the experience is probably supported by AI. And scientists are applying AI methods to find indications of disease, to design experiments, and to make discovery processes more efficient.

The National Institutes of Health (NIH) has been using AI to improve science and health, too, but it’s also using AI in other ways.

Earlier this fall, the White House Office of Science and Technology Policy hosted a summit to highlight ways that the Federal Government uses AI to achieve its mission and improve services to the American people. I was proud to represent NIH and provide examples of how AI is being used to make NIH more effective and efficient in its work.

For example, each year NIH faces the challenge of assigning the more than 80,000 grant applications it receives to the proper review group.

Here’s how the process works now:  Applications that address specific funding opportunity announcements are assigned directly by division directors. Then the Integrated Review Groups (clusters of study sections grouped around general scientific areas) assign the applications to the correct division or scientific branch. A triage officer handles assignments without an identified liaison. This process takes several weeks and may involve passing an application through multiple staff reviews.

Staff at NIH’s National Institute of General Medical Sciences (NIGMS) creatively addressed this challenge by developing and deploying natural language processing and machine learning to automate the process for their Institute. This approach uses a machine learning algorithm, trained on historical data, to find a relationship between the text (title, abstract, and specific aims) and the scientific research area of an application. The trained algorithm can then determine the most likely scientific area of a new application and automatically assign it a program officer who is a subject matter expert in that area.

The new process works impressively well, with 92% of applications referred to the correct scientific division and 84% assigned to the correct program officer, matching the accuracy rate routinely achieved by manual referrals. This change has resulted in substantial time savings, reducing the process from two to three weeks to less than one day. The new approach ensures the efficient and consistent referral of grant applications and liberates program officers from the labor-intensive and monotonous manual referral process, allowing them to focus on higher-value work. It even allows for related institutional knowledge to be retained after staff departures. NIGMS is currently working with the NIH electronic Research Administration (eRA) to incorporate the process into the enterprise database for NIH-wide use.

Now for a second example that’s more pertinent to NLM.

Our PubMed repository receives over 1.2 million new citations each year, and over 2.3 million people conduct about 2.5 million searches using PubMed every day. An average query returns hundreds to thousands of results presented in reverse chronological order of the date the record is added. Yet our internal process-monitoring determined that 80% of the people using PubMed do not go beyond the first page of results, a behavior also seen in general web searches. This means that even if a more relevant citation is on page 4 or page 18, the user may never know.

Zhiyong Lu, PhD and his team from NLM’s National Center for Biotechnology Information applied machine learning strategies to improve the way PubMed presents search results. Their goals were to increase the effectiveness of PubMed searches by helping users efficiently find the most relevant and high-quality information and improve usability and the user experience through a focus on the literature search behaviors and needs of users. Their approach is called the Best Match algorithm, and the technical details can be found in a paper by Fiorini N, Canese K, Starchenko G, et al., PLoS Biol. 2018.

The Best Match algorithm works like this:  In preparation for querying, all articles in the PubMed repository are tagged with key information and metadata, including the publication date, and with an indicator of how often the article has been returned and accessed by previous searches, as part of a model-training process called Learning-to-Rank (L2R). Then, when a user enters a query phrase in the search box on the PubMed website, the phrase is mapped using the PubMed syntax, and the search is launched. In a traditional search, the results are selected based on keyword matching and are presented in reverse chronological order. Through Best Match, the top 500 results—returned via a classic term-weighting algorithm—are re-sorted according to dozens of features of the L2R algorithm, including the past usage of an article, publication date, relevance score, and type of article. At the top of the page, the search results are clearly marked as being sorted by “Best Match.”

This new approach will become the core of a new implementation of PubMed, due out by the spring of 2020.

In addition to the examples I described above, NIH is exploring other ways to use AI. For example, AI can help determine whether the themes of research projects align with the stated priorities of a specific Institute, and it can provide a powerful tool to accelerate government business practices. Because of AI’s novelty, NIH engages in many steps to validate the results of these new approaches, ensuring that unanticipated problems do not arise.

In future posts, I look forward to sharing more about how NIH is improving operations through innovative analytics.