FDA Budget Matters: A Cross-Cutting Data Enterprise for Real World Evidence

By: Scott Gottlieb, M.D.

Over time, as our experience with new medical products expands, our knowledge about how best to maximize their benefits and minimize any potential risks, sharpens with each data point we gather. Every clinical use of a product produces data that can help better inform us about its safety and efficacy.

Dr. Scott GottliebThe FDA is committed to developing new tools to help us access and use data collected from all sources. This includes ways to expand our methodological repertoire to build on our understanding of medical products throughout their lifecycle, in the post market. We don’t limit our knowledge to pre-market information, traditional de novo post-market studies, and passive reporting. Newer methodologies enable us to collect data from routine medical care and develop valid scientific evidence that’s appropriate for regulatory decision making to help patients and health care providers prevent, diagnose, or treat diseases.

This includes our ability to leverage what’s often referred to as “real world data.” Real world data consists of data relating to patient health status and/or the delivery of health care routinely collected from a variety of sources, including information obtained at the point of care. By using this information, we can gain a deeper understanding of a medical product’s safety and benefits, its additional treatment implications, and its potential limitations. By better leveraging this information, we can also enable more efficient medical product development by integrating greater complements of safety and benefit information gleaned from clinical care. This is especially true when it comes to our important obligation to continue to evaluate products in the post-market setting.

Traditional randomized clinical trials can provide key information on a medical product’s performance to support regulatory marketing decisions and health care decisions made by patients and providers. However, traditional clinical trials have their own limitations. The FDA, along with others, sometimes benefit from more information than these trials can provide about how medical products are used in medical practice.

For example, traditional clinical trials have patient inclusion and exclusion criteria that often narrow the patient population that can participate in a traditional trial. So, patients who’ve undergone another treatment, or who are taking other medications, may not qualify for a certain trial that’s looking for patients who haven’t been treated for that disease or condition, or who are taking certain medications.

When this product comes to market, it’s possible that patients who pursued other treatments or patients taking medications for other conditions will be prescribed this therapy. Because these patients weren’t studied, there’ll be no clinical trial evidence available showing how these other factors may affect the safety or efficacy of this product. Clinical trials provide a picture of a medical product’s potential in a narrow and highly controlled setting. But they do not provide a complete picture as to how a product works outside of that setting. This can limit our broader understanding of how a new product will work in “the real world.”

Real World Evidence diagramThe FDA is uniquely positioned and qualified to lead the effort to expand the use of real world data to address these knowledge gaps. Over the past decade, through the FDA’s Sentinel System and the National Evaluation System for health Technology (NEST), the FDA has begun to harness formerly untapped information to help us answer some of the most pressing questions facing patients and providers about the use of medical products. This use of real world data is referred to as “real world evidence.” This is meant to express the use of real world data to generate practical clinical evidence regarding the potential benefits or risks of a product. In this case, the evidence is derived from analysis of real world data.

We’re working to promote and expand the use of both real world data and real world evidence in medical product development and regulatory science. And not only for FDA uses, but also for others that seek to answer critical questions about health care delivery. To accomplish this goal, the FDA will leverage our knowledge and skills from building and using the Sentinel System and further supporting the development of NEST. Most importantly, we must develop the means to govern the responsible use of these data and to provide timely access to a broad group of public and private entities through the creation of a national resource. All the while, we must maintain strict data security and privacy of personal information.

To these ends, as part of the President’s Fiscal Year 2019 Budget, we’ve put forward a $100M medical data enterprise proposal to build a modern system that would rely on the electronic health records from about 10 million lives. This system would expand the data enterprise that we already maintain by incorporating new information from electronic health records, and other sources that would allow us to more fully evaluate medical products in the post-market setting.

This is the next evolution in the Agency’s development of a comprehensive data enterprise to improve medical product regulation and better inform us on the safety and benefits of new innovations.

Post-Market Data Sources: Claims Data vs. EHRs

Previously, our investments in post-market data have mostly focused on the development of systems to consolidate and analyze information derived from healthcare payer claims. This was a key advance in our regulatory system. And relying on health claims information was the state of the art at the time that we built these systems. Now we have the capacity to use clinical data derived from electronic health records to develop faster reporting on the performance of medical products in real world medical settings.

Claims data provides important insights. But it also has some limitations. For example, there’s an inherent lag between when a medical event occurs, and when it’ll show up in payer claims. There’s also some ambiguity in this process. It’s not always clear, by looking at claims data alone, what actually happened to the patient and whether the medical product was a factor. So, in the current system, we need to make certain assumptions when we evaluate claims data, to draw conclusions from this information. And some of these assumptions can inject uncertainty. The FY 2019 Budget request seeks to address some of these limitations by giving the Agency the ability to access the clinical medical information contained in de-identified electronic health records.

Investments in such a system can become a national utility for improving medical care, and allowing the FDA to optimize its regulatory decisions. It would give patients and providers the access to near-real-time, post-market information that can better inform their decisions. Such an enterprise can not only support our evaluation of safety and benefit using data derived from real-world settings, but it can also make the development of new innovations more efficient. If we have more dependable, near-real-time tools for evaluating products in real-world settings, we can allow key questions to be further evaluated in the post-market setting. This can allow some of the cost of development to be shifted into the post-market, where we can sometimes access better information about how products perform in real-world settings.

Establishing a System that can Leverage All Data Sources

Real world data can come from many sources. It not only can include electronic health records, but also claims and billing activities, product and disease registries, patient-related activities in out-patient or in-home use settings, and mobile health devices. It’s key that the sources of these data elements, such as different health care systems, be able to communicate electronically. This requires full “interoperability” and the elimination of any silos. The FY 2019 Budget request seeks to establish these building blocks, and assemble the data into an interoperable platform. There are several foundational steps that we’re already undertaking to build a strong programmatic basis for using real world data and evidence.

Achieving interoperability and establishing data standards, while conceptually obvious, is by no means easy to accomplish. Different groups may collect the same information in different ways. Consider that one group collects temperature using Celsius and another uses Fahrenheit. The group that uses Celsius may document a temperature of 37 degrees, while the one that uses Fahrenheit would document a temperature of 98.6 degrees. While these both are the same finding, in the absence of data standards, they would appear drastically different. Therefore, one key to this effort is the development of data standards and agreed upon definitions that allow different groups to meaningfully share their data.

Additionally, as noted above, there are many potential sources of real world data. Our familiarity and ability to harness these data varies across these sources. For example, the Sentinel System has taken advantage of a well-established source of real world data, claims and billing data. But claims and billing data, while well established and characterized, don’t necessarily capture the full scope of actual patient treatment. When it comes to medical devices, these claims data may not include the Unique Device Identifier which can limit the utility of the information. In addition, physicians may not be recoding every treatment in claims and billing data because of payment bundles, so the exact treatment is not known.

In comparison, electronic health records capture more of the patient experience and have the potential to provide more “real-time” information. But the information is also captured in a much less standardized way. Often key information is documented in unstructured ‘free text’ as part of a provider’s note. So, standardizing this information — and assembling it into formats that can allow for easier analysis and integration — will take additional investment in systems that can consolidate this information and make it interoperable.

Part of our proposed investment will go toward building these new capabilities to assemble real world data into formats to make this information more accessible. Ultimately, our goal is that such a tool can become a national utility that can be accessed by qualified research partners to inform a host of important clinical questions.

Improving Clinical Trials

The development of such a tool can also make the entire clinical trial process much more efficient. And it can enable us to enroll more patients from more diverse backgrounds into trials.

For example, real world data can be used to more efficiently identify and recruit patients for a clinical trial. Key design considerations, such as randomization, can be integrated across clinical care settings, introducing a much more diverse population into the clinical trial system. Innovative statistical approaches — such as Bayesian and propensity scores methods — can combine information from different sources and potentially reduce the size and duration of a clinical trial while expanding the scope of healthcare questions that we’re able to evaluate. This will enable a modern clinical trial system that improves upon trials being conducted in large medical care centers. It could enable more clinical trials at smaller community-based health care providers. Such a system can expand the number of patients we’re able to evaluate, and broaden the information that we’re able to collect, while at the same time reducing the cost of developing this information. We can have more and better information, and a less costly process.

All of this is contingent upon our ability to have confidence in the quality of data we’re accessing to make decisions, be that regulatory or derived from individual patient care.  We’re working with public and private partners to ensure optimal data quality, validity, and utilization. Our goal is to develop better data standards, to promote interoperability, and improve data quality.

Investing in Tools to More Wisely Use Data to Improve Health

Data quality has different impacts when considering the use of this data for individual patient care as opposed to broader public health evaluations. However, our capacity to make effective use of real world data and real world evidence will have a profound impact on individual patients and the public health.

Investing in the creation of a national resource that leverages real world data, establishes data standards to facilitate interoperability, and promotes data quality for the integration of this evidence into medical product development and clinical care is a key national investment. It’ll improve patient care, and make the process for developing safe and effective new medical innovations more efficient. It’ll give us a near real-time tool for monitoring the post-market safety of medical products, and will help inform better and more timely regulatory decisions.

Most importantly, such a system will provide patients with better care and more informed treatment decisions. The wider use of real world data could decrease the cost of product development, while increasing our understanding of how, when, and in whom, to use medical products. It’ll allow us to use the post-market period to refine our understanding of medical products. And it’ll allow us to make reliable post-market information available to providers and patients to better inform their treatment decisions.

Scott Gottlieb, M.D., is Commissioner of the U.S. Food and Drug Administration 

Follow Commissioner Gottlieb on Twitter @SGottliebFDA

What We Mean When We Talk About Data

By: Robert M. Califf, M.D. and Rachel Sherman, M.D., M.P.H.

Robert M. Califf, M.D., MACC, FDA's Commissioner of Food and Drugs

Robert M. Califf, M.D., Commissioner of the U.S. Food and Drug Administration

Medical care and biomedical research are in the midst of a data revolution. Networked systems, electronic health records, electronic insurance claims databases, social media, patient registries, and smartphones and other personal devices together comprise an immense new set of sources for data about health and healthcare. In addition, these “real-world” sources can provide data about patients in the setting of their environments—whether at home or at work—and in the social context of their lives. Many researchers are eager to tap into these streams in order to provide more accurate and nuanced answers to questions about patient health and the safety and effectiveness of medical products—and to do so quickly, efficiently, and at a lower cost than has previously been possible.

But before we can realize the dramatic potential of the healthcare data revolution, a number of practical, logistical, and scientific challenges must be overcome. And one of the first that must be tackled is the issue of terminology.

Defining Terms

Although “data,” “information,” and “evidence” are often used as if they were interchangeable terms, they are not. Data are best understood as raw measurements of some thing or process. By themselves they are meaningless; only when we add critical context about what is being measured and how do they become information. That information can then be analyzed and combined to yield evidence, which in turn, can be used to guide decision-making. In other words, it’s not enough merely to have data, even very large amounts of it. What we need, ultimately, is evidence that can be applied to answering scientific and clinical questions.

So far, so good. But what do we mean when we talk about “real-world data” or “real-world evidence”?

Rachel Sherman

Rachel Sherman, M.D., M.P.H., FDA’s Associate Deputy Commissioner for Medical Products and Tobacco.

Clinical research often takes place in highly controlled settings that may not reflect the day-to-day realities of typical patient care or the life of a patient outside of the medical care system. Further, those who enroll in clinical trials are carefully selected according to criteria that may exclude many patients, especially those who have other diseases, are taking other drugs, or cannot travel to the investigation site. In other words, the data gathered from such studies may not actually depict the “real world” that many patients and care providers will experience—and this could lead to important limitations in our understanding of the effectiveness and safety of medical treatments. Clinicians and patients must be able to relate the results of clinical trials—studies that are done in controlled environments with certain patient populations excluded and which may therefore be challenging to generalize—to their own professional and personal experiences. It seems straightforward, then, to think that studies including a much fuller and more diverse range of individuals and clinical circumstances could ultimately lead to better scientific evidence for application to decisions about use of medical products and healthcare decisions.

But “real-world evidence” has its own issues that must be understood and dealt with carefully. First of all, the vague term “real-world” may imply a closer relationship with the truth—that the real-world measurement is preferable to one taken in a controlled environment. For example, is “real-world” blood pressure data gathered from an individual’s personal device or health app better (e.g., more reliable and accurate) than a blood pressure measurement from a doctor’s office? It could be, because a patient’s blood pressure might be uncharacteristically elevated during a visit to the physician. But at the same time, do we know enough about the data gathered from the patient’s personal device—how accurate is it? Is the patient taking their own blood pressure correctly? What other factors might be affecting it?—to use it for generating evidence? Already we are being reminded of the complexities of potentially relying on data that were gathered for purposes other than the ones for which they were originally intended.

In most cases “real-world evidence” is thought of as reflecting data already collected, i.e., epidemiologic or cohort data that researchers review and analyze retrospectively. Also of interest is whether randomized trials can be conducted in these “real-world” environments. In considering comparisons of treatments, one must always consider the possibility that the treatments were not assigned randomly, but reflected some relevant patient characteristic. This is, of course, the reason for doing randomized clinical trials.

Better Terms for Complex Subjects

There is little doubt that the new sources of data now being opened to researchers, clinicians, and patients hold enormous potential for improving the quality, safety, and efficiency of medical care. But as we work to understand both the promise and pitfalls of far-reaching technological changes, we need a more functional vocabulary for talking about these complex subjects, one that allows us to think about data, information, and evidence in ways that capture multiple dimensions of quality and fitness for purpose (e.g., for appropriate use in regulatory decision making). The incorporation of “real-world evidence”—that is, evidence derived from data gathered from actual patient experiences, in all their diversity— in many ways represents an important step toward a fundamentally better understanding of states of disease and health. As we begin to adapt “real-world data” into our processes for creating scientific evidence, and as we begin to recognize and effectively address their challenges, we are likely to find that the quality of the answers we receive will depend in large part on whether we can frame the questions in a meaningful way.

Robert M. Califf, M.D., previously FDA’s Deputy Commissioner for Medical Products and Tobacco, became FDA’s Commissioner of Food and Drugs on Feb. 25, 2016.

Rachel Sherman, M.D., M.P.H., is FDA’s Associate Deputy Commissioner for Medical Products and Tobacco.