Principles for the use of large-scale medical databases to generate real-world evidence

ABSTRACT

BACKGROUND

The development and expanding accessibility of large-scale medical databases support increasing use of these databases for clinical epidemiological studies in Japan.

METHODS

Members of the Database Utilization Committee of the Society of Clinical Epidemiology discussed the ways to enhance scientific rigor in the utilization of large-scale medical databases in Japan. The topics covered included what we need to know about the current research environment when conducting large-scale medical database studies, the key points to consider when conducting clinical research using these databases, and the use of real-world evidence in regulatory decision making.

RESULTS

In generating real-world evidence, several evaluation principles should be noted: the characteristics of data commonly used for clinical epidemiology in Japan; the need for reproducibility and transparency in database studies; the development of research questions for database studies; the need to assess the quality and validity of data; and legal and ethical considerations. We also introduce the discussions on real-world evidence for drug approval applications.

CONCLUSION

With rapid technological developments, the characteristics of medical databases and analytical methods will evolve over time. Keeping these principles in mind, we must continuously update the evaluation methods and strategies used to assess the data.

INTRODUCTION

The development and expanding accessibility of large-scale medical databases have supported the rapid increase in clinical epidemiological studies based on such databases in Japan. Examples of these databases include the nationwide Diagnostic Procedure Combination (DPC) databases, commercial and regional claims databases, nationwide patient-, disease-, and procedural-registries, and electronic medical record (EMR) databases. The aims of this review, developed by the Database Utilization Committee of the Society for Clinical Epidemiology, are to summarize the current environment surrounding clinical research studies using large-scale medical databases in Japan, and to introduce some key principles that should be considered when conducting such studies.

DATABASES FOR CLINICAL EPIDEMIOLOGICAL STUDIES

Two common types of databases currently used for observational clinical research are administrative claims databases and EMR databases. Administrative claims databases are archives of medical insurance bills generated by medical facilities, including hospitals, clinics, and pharmacies. Hospitals and clinics prepare medical insurance claims and send them to the corresponding insurers through Health Insurance Claims Review & Reimbursement Services. Pharmacies also prepare prescription insurance claims. Since 2003, Japan has adopted a diagnosis-related group-based payment system for acute inpatient care, known as the DPC payment system, and majority of acute-care hospitals currently prepare DPC claims [1]. The DPC data are created from the DPC claims records of hospitals, and contain records of disease diagnoses, prescriptions, and procedure claims. DPC data also include some clinical information on patients’ characteristics (e.g., height, weight, smoking status) and disease severity (e.g., the A-DROP pneumonia classification and the Union for International Cancer Control [UICC] staging of neoplasms) depending on the case. Because the DPC database is mainly prepared for inpatient claims, it contains limited data for outpatient care, although the database is linkable to outpatient claims files if available. The number of researchers using the DPC database is increasing, resulting in an increase in scientific publications in the past decade [2–6]. One drawback of the DPC database is that it cannot link patients across hospitals. When a patient is admitted to a different hospital, he/she will be considered a different patient within the database, which limits the usefulness of DPC data for longitudinal follow-up studies.

Pharmacy claims databases are created from pharmacy dispensation claims [7, 8]. Medical claims contain prescription information, but it is unclear whether the patients’ prescriptions are dispensed at pharmacies. Dispensation claims should more accurately capture drug usage by patients than prescription claims, although the true drug intake is still not necessarily known. A limitation of pharmacy claims databases is that they do not contain information on disease diagnoses or procedures. Furthermore, like DPC data, the pharmacy claims databases cannot link patients across different pharmacies, so their usefulness in longitudinal follow-up studies is also limited.

Administrative claim databases collected by insurers contain medical claims, prescription claims, and DPC claims [7, 8]. These databases contain patient IDs, which allow claims to be linked across different medical facilities, and researchers can follow the patients’ healthcare utilization across different healthcare providers. Therefore, these databases are suitable for research requiring longitudinal follow-up [9–11]. However, the general limitations of research studies based on administrative claims databases still apply, including the accuracy of diagnostic claims and lack of detailed clinical information.

EMR databases contain clinical information that is captured in the electronic records at healthcare facilities. Most EMR databases also contain medical claims and DPC claims from these facilities, in addition to clinical information extracted from the medical information system. Examples of clinical information include laboratory test results, diagnostic imaging results, pathology results, and reports of patients’ symptoms (e.g., numeric pain rating scales). The validity of the output from clinical epidemiology studies based on EMR databases could be higher than that of studies that only use claims databases because these detailed clinical data are included [12]. However, there are multiple EMR software systems in Japan, and it is challenging to aggregate data from different EMR systems.

USE OF LARGE-SCALE DATABASES TO GENERATE REAL-WORLD EVIDENCE

Analyzing real-world databases (RWDs), such as health claims databases, EMR databases and disease or procedural registries, can be a powerful way to understand the natural history of diseases and current medical practices. These data can also provide information on how patients adhere to their treatment regimens, and allow researchers to estimate the impact of treatments in real-world settings, thus generating real-world evidence. RWDs typically include large number of patients or participants and even a small difference can be interpreted as statistically significant [13]. However, the direct utilization of claims or EMR data, which are not collected for research purposes, may have uncertain accuracy. Therefore, the general principles of observational studies must be carefully considered when conducting RWD studies and interpreting their results.

One of the key questions to be asked before conducting a database study is “who is included in the database?”. Researchers should carefully consider whether the subjects included in the database are appropriate and reasonable targets for the purpose of the study. For example, a health claims database derived from an employer-based insurance scheme in Japan contains no data of the subjects aged ≥75 years because these subjects are covered by ‍the later-stage elderly healthcare system. If the study question allows us to conduct the study among subjects aged <75 years, we should generalize the findings derived from the database to the elderly population aged ≥75 years with caution.

The validity of the data must also be considered, especially when they are taken from claims-based databases. The validity of exposure and outcomes data, such as those for medical procedures or diseases of interest, must be assessed carefully. When a disease is identified using definitions based solely on diagnostic codes, it may include patients with real diseases, as well as patients with suspected conditions based on differential diagnoses. It is also possible that some patients are only allocated these diagnostic codes to justify certain examinations or medications. Distinguishing a correct diagnosis of the disease from a pool of suspected disease diagnoses on the database based only on diagnostic codes is challenging; it requires both careful examination of the claims records and the recording patterns in the specific database, together with an assessment of how physicians are practicing medicine in their clinics. Validation studies of disease definitions should be conducted whenever possible, although this is very costly in many situations. The number and scope of validation studies of administrative databases are still limited in Japan, but are increasing [14–17]. It may be more accurate to develop definitions of exposure and outcomes that are not based only on simple disease diagnostic codes but also incorporating other supporting data, such as medical procedures and prescriptions.

REPRODUCIBILITY AND TRANSPARENCY IN OBSERVATIONAL STUDIES USING LARGE-SCALE MEDICAL DATABASES

In 2017, the International Committee of Medical Journal Editors (ICMJE) announced that it will require a data-sharing statement for all participant data from clinical trials at manuscript submission [18]. This is necessary to ensure the reproducibility of research. Researchers are now required to disclose to the public the data they use for research. However, in research based on large-scale medical databases in Japan, it is often difficult for researchers to share these data because patient privacy must be protected and data usage is strictly regulated [19]. Therefore, it is even more important for researchers to ensure their studies are reproducible and transparent.

In reporting guidelines for research based on large-scale medical databases published in 2017, Wang and colleagues recommended that researchers clearly report the following processes to ensure their research is reproducible [19]: 1) data extraction, 2) analytic data creation, and 3) data analysis. The National Database (NDB) and DPC database, two of the major nationwide large-scale medical databases in Japan, store data according to the record specifications established by the Ministry of Health, Labor and Welfare (MHLW) of Japan. To organize electronic medical record information, the MHLW developed SS-MIX, a system of standard record specifications and standard data storage methods that can be applied across different medical institutions [20]. However, the subsequent data extraction, tabulation, and analytical procedures differ across researchers. Unfortunately, at this time, very few research papers based on national large-scale medical databases fully report the processes used, including the data extraction process.

To make the data extraction process more transparent, it is important to define and use a standard data-processing procedure. A previous study reported the development of a common data mart (CDM) that can be used for various types of analyses [21]. Using the CDM, researchers can convert raw longitudinal data to CDM and generate an analysis table [21]. In Japan, Kuroda and others are developing a general-purpose data mart for research related to the NDB [22]. Because these data marts require continuous maintenance, the continuous involvement of nonprofit entities, such as academic societies, is needed.

RESEARCH QUESTIONS FOR LARGE-SCALE MEDICAL DATABASE RESEARCH

It is essential to develop a simple, clear, and pre-declared research question for all database studies. This clear research question usually involves four factors: patients, exposure, comparison, and outcome. Each factor must be clearly defined by the variables included in the database. For example, let us consider the “patients” required for a research question related to diabetes. The selected patients would differ if we defined diabetes by laboratory values for HbA1c or if we defined diabetes based on diagnostic codes for diabetes recorded in their claims. Unless each factor is appropriately defined based on the purpose of the research question, the results of the database research cannot be properly interpreted.

Another important point for developing a research question for database research is the meaning of the research question itself. Database research may start with an overview of many variables in the database, and changing the combination of variables used may give rise to new research questions in some cases. Consequently, there is a risk that a large quantity of results will be produced without sufficient examination of their clinical meaning. Of course, data-driven exploratory analyses may be useful for forming a hypothesis in some cases, but these analyses must be clearly distinguished from hypothesis-based analyses.

The following criteria can be considered when checking whether the requirements of a good research question for database research are met: 1) the question clearly shows how the four factors (patient, exposure, comparison, and outcome) are defined; 2) the question clearly shows how to reproduce the same dataset and results from the raw data; and 3) the question clearly shows the clinical importance of the research question based on previous studies and clinical knowledge.

DATA QUALITY AND ASSESSMENT

The validity of the evidence generated from database studies is highly dependent on the quality of the data, which must be assessed and validated. When discussing the quality of data, especially data from registry databases, two important characteristics are their completeness and their accuracy. Completeness is achieved when cases that should be included in the database are included, without duplication or restriction. For instance, it was previously reported that the registration of some cases with poor outcomes was omitted from a registry database [23]. This would introduce some bias if the data are used to estimate the event rates or mortality associated with the relevant procedures.

In contrast, the accuracy of data can be assessed by examining whether there are any errors in the data for each subject compared with the medical records at the reporting facility (source document). When evaluating the accuracy of data, it is important to check the concordance of information entered into the database with the source data, and assess how any discordance occurred (e.g., whether the discordance is due to a failure to record a true event, or whether there is a false record of a nonexistent event) to allow researchers to estimate the sensitivity and specificity of the data. Previous studies on the data accuracy for nationwide registries have reported that although hard endpoints, such as 30-day and 90-day mortality, are recorded with high accuracy, postoperative complications may be less accurately recorded [24–27]. However, for most complications, the estimated specificities were high, supporting their use for outcome measures in comparative safety studies, assuming no differential accuracy between the two groups being compared. Although validation studies are essential, important factors such as complications may not be clearly documented in a patient’s medical record, making the process of auditing and validation even more difficult and costly.

LEGAL AND ETHICAL CONSIDERATIONS

The importance of legal or ethical considerations in database research was recently noted. For example, the Taipei Declaration adopted by the World Medical Association in 2016 [28] clearly stipulates the ethics of database research. Some public databases, such as the NDB, have a legal basis for research use, but there is no general regulation of database research in Japan [29, 30]. Japan’s new Clinical Research Act, enforced in April 2018, does not include observational studies among its targets. The Protection of Personal Information Act (PPI Act), which regulates the handling of personal information, excepts research in general. Only the National Research Ethics Guidelines [31] set such rules, and based on these guidelines, only research approved by an Ethics Review Board can be conducted. These guidelines basically impose strict rules that override the PPI Act. In observational research using databases, opt-in is required in principle, as specified by the PPI Act.

A new law enacted in 2018 [32], usually referred to as the Next-Generation Medical Infrastructure Act, allows the use of anonymized medical data in an opt-out basis. Database research using anonymized medical data according to this new law is expected to accelerate.

For genomic data, the Genomic Research Guideline [33], together with some references in the PPI Act, applies stricter rules than other medical research guidelines [34]. With the increasing importance of international data sharing, including genomic data, the need for global regulation of the use of these data is increasing [35].

USE OF RWD FOR DRUG APPROVAL APPLICATIONS

Although the double-blind randomized controlled trial (RCT) is the gold standard approach to assess the efficacy of a medication, and is therefore usually considered by the regulators for decision making, it may not be feasible for ethical reasons, or because of the high costs involved or the limited generalizability of the findings in some situations. These limitations of RCTs have been repeatedly noted [36, 37].

The use of real world evidence in regulatory decision making is increasingly discussed in Japan and other countries. In the USA, where the 21^st Century Cures Act [38] tasks the Food and Drug Administration (FDA) the evaluation of the use of RWE for such purpose, FDA has released a framework for their RWE program [39]. While the use of RWD for monitoring and evaluating the safety of drug products in the postmarket phase, exemplified by the Sentinel system [40], has been conducted for a number of years, the framework also discusses the potential use of RWE for evaluating effectiveness of drugs. In fact, applications for indication expansion for a drug that included data from EMR-based database and a claims database have been submitted to and reviewed by the FDA, and a summary of their assessment has been published [41]. In Japan, with the revised Good Post-marketing Study Practice ordinance put into effect in April 2018, the use of database-based studies as part of post marketing study activities has become possible. Further collaboration among the regulators, academia and industry can boost the discussion on this topic and will enable developments of the rules and guidelines for scientifically appropriate use of RWD for RWE generation, which will strengthen Japan’s competitiveness in the field.

CONCLUSION

With rapid technological developments, the characteristics of medical databases and the analytical methods for their use will evolve over time. Keeping in mind the principles discussed here, we must continuously update the evaluation methods and strategies used to assess such databases.

DISCLOSURE

Hiraku Kumamaru, Arata Takahashi and Hiroaki Miyata are affiliated with the department of Healthcare Quality assessment at the University of Tokyo. The department is a social collaboration department supported by National Clinical Database, Johnson & Johnson K.K., and Nipro corporation. Hironobu Tokumasu is Chief Operating Officer of the Real World Data Co., Ltd. Seigo Hara is CEO and a shareholder of MICIN.inc. Kotonari Aoki is an employee of Chugai Pharmaceutical Co, Ltd.

REFERENCE

• 1.   Ishikawa  KB. Medical Big Data for Research Use: Current Status and Related Issues. Japan Med Assoc J 2016;59:110–124.
• 2.   Fukuda  T,  Imai  S,  Nakadera  M,  Wagatsuma  Y,  Horiguchi  H. Postoperative daily living activities of geriatric patients administered general or spinal anesthesia for hip fracture surgery: A retrospective cohort study. J Orthop Surg 2018;26:1–9.
• 3.   Tagami  T,  Matui  H,  Fushimi  K,  Yasunaga  H. Validation of the prognostic burn index: a nationwide retrospective study. Burns 2015;41:1169–1175.
• 4.   Shinjo  D,  Matsumoto  K,  Terashima  K,  Takimoto  T,  Ohnuma  T,  Noguchi  T, et al. Volume effect in paediatric brain tumour resection surgery: analysis of data from the Japanese national inpatient database. Eur J Cancer 2019;109:111–119.
• 5.  Nakajima M,  Aso  S,  Matsui  H,  Kasqynski  RH,  Fushimi  K,  Yamaguchi  Y, et al. Prevalence of myocardial injury requiring percutaneous coronary intervention after acute carbon monoxide poisoning. Eur J Emerg Med 2019;Epub ahead of print.
• 6.   Hayakawa  I,  Okubo  Y,  Nariai  H,  Michihata  N,  Matsui  H,  Fushimi  K, et al. Recent treatment patterns and variations for pediatric acute encephalophathy in Japan. Brain Dev 2019;Epub ahead of print.
• 7.   Shimizu  E,  Kawahara  K. Assessment of Medical Information Databases to Estimate Patient Numbers. Jpn J Pharmacoepidemiol 2014;19:1–11.
• 8.   Tanaka  S,  Seto  K,  Kawakami  K. Pharmacoepidemiology in Japan: medical databases and research achievements. J Pharm Heal Care Sci 2015;1:1–4.
• 9.   Okumura  Y,  Tachimori  H,  Matsumoto  T,  Nishi  D. Exposure to psychotropic medications prior to overdose: a case-control study. Psychopharmacology (Berl) 2015;232:3101–3109.
• 10.   Kasai  S,  Sakai  R,  Koike  R,  Kohsaka  H,  Miyasaka  N,  Harigai  M. Higher risk of hospitalized infection, cardiovascular disease, and fracture in patients with rheumatoid arthritis determined using the Japanese health insurance database. Mod Rheumatol 2019;29:788–794.
• 11.   Hiroi  S,  Shimasaki  Y,  Kikuchi  T,  Otsuka  Y,  Iwasaki  K,  Ohishi  M. Analysis of second- and third-line antihypertensive treatments after initial therapy with an angiotensin II receptor blocker using real-world Japanese data. Hypertens Res 2016;39:907–912.
• 12.   Takeuchi  Y,  Shinozaki  T,  Kumamaru  H,  Hiramatsu  T,  Matsuyama  Y. Analyzing intent-to-treat and per-protocol effects on safety outcomes using a medical information database: an application to the risk assessment of antibiotic-induced liver injury. Expert Opin Drug Saf 2018;17:1071–1079.
• 13.   Corrao  G. Building reliable evidence from realworld data: Methods, cautiousness and recommendations. Epidemiol Biostat Public Heal 2013;10:1–40.
• 14.   Yamana  H,  Moriwaki  M,  Horiguchi  H,  Kodan  M,  Fushimi  K,  Yasunaga  H. Validity of diagnoses, procedures, and laboratory data in Japanese administrative data. J Epidemiol 2017;27:476–482.
• 15.   Ando  T,  Ooba  N,  Mochizuki  M,  Koide  D,  Kimura  K,  Lee  SL, et al. Positive predictive value of ICD-10 codes for acute myocardial infarction in Japan: A validation study at a single center. BMC Health Serv Res 2018;18:1–7.
• 16.   Sato  I,  Yagata  H,  Ohashi  Y. The Accuracy of Japanese Claims Data in Identifying Breast Cancer Cases. Biol Pharm Bull 2015;38:53–57.
• 17.   Imai  S,  Yamana  H,  Inoue  N,  Akazawa  M,  Horiguchi  H,  Fushimi  K, et al. Validity of administrative database detection of previously resolved hepatitis B virus in Japan. J Med Virol 2019;91:1944–1948.
• 18.   Taichman  DB,  Sahni  P,  Pinborg  A,  Peiperl  L,  Laine  C,  James  A, et al. Data Sharing Statements for Clinical Trials—A Requirement of the International Committee of Medical Journal Editors. N Engl J Med 2017;376:2277–2279.
• 19.   Wang  SV,  Schneeweiss  S,  Berger  ML,  Brown  J,  Vries F  de,  Douglas  I, et al. Reporting to Improve Reproducibility and Facilitate Validity Assessment for Healthcare Database Studies V1.0. Value Heal 2017;20:1009–1022.
• 20.   Kimura  M,  Nakayasu  K,  Ohshima  Y,  Fujita  N,  Nakashima  N,  Jozaki  H, et al. SS-MIX: A Ministry Project to Promote Standardized Healthcare Information Exchange. Methods of Information in Medicine. Methods Inf Med 2011;50:131–139.
• 21.   Overhage  JM,  Ryan  PB,  Reich  CG,  Hartzema  AG,  Stang  PE. Validation of a common data model for active safety surveillance research. J Am Med Inform Assoc 2012;19:54–60.
• 22.  NTTdata. National Database [Internet]. [cited 2019 May 12]. Available from: https://www.nttdata.com/jp/ja/case/2018102501
• 23.   Elfstrom  J,  Stubberod  A,  Troeng  T. Patients not included in medical audit have a worse outcome than those included. Int J Qual Heal Care 1996;8:153–157.
• 24.   Kneuertz  PJ,  Pitt  HA,  Bilimoria  KY,  Smiley  JP,  Cohen  ME,  Ko  CY, et al. Risk of morbidity and mortality following hepato-pancreato-biliary surgery. J Gastrointest Surg 2012;16:1727–1735.
• 25.   Epelboym  I,  Gawlas  I,  Lee  JA,  Schrope  B,  Chabot  JA,  Allendorf  JD. Limitations of ACS-NSQIP in reporting complications for patients undergoing pancreatectomy: underscoring the need for a pancreas-specific module. World J Surg 2014;38:1461–1467.
• 26.   Tomotaki  A,  Kumamaru  H,  Hashimoto  H,  Takahashi  A,  Ono  M,  Iwanaka  T, et al. Evaluating the quality of data from the Japanese National Clinical Database 2011 via a comparison with regional government report data and medical charts. Surg Today 2018;49:65–71.
• 27.   Takahashi  A,  Kumamaru  H,  Tomotaki  A,  Matsumura  G,  Fukuchi  E,  Hirata  Y, et al. Verification of Data Accuracy in Japan Congenital Cardiovascular Surgery Database Including Its Postprocedural Complication Reports. World J Pediatr Congenit Hear Surg 2018;9:150–156.
• 28.  World Medical Association; Declaration of Taipei [Internet]. [cited 2019 Nov 11]. Available from: https://www.wma.net/what-we-do/medical-ethics/declaration-of-taipei/
• 29.   Yamamoto  R. Large-scale Health Information Database and Privacy Protection. JMAJ 2016;59:91–109.
• 30.   Yamamoto  H. Use of personal information in medical research in Japan. Lancet 2016;388:1981–1982.
• 31.  Ministry of Education, Culture, Sports, Science and Technology & Ministry of Heath L and W. Ethical guidelines for medical and health research involving human subjects. [Internet]. [cited 2019 Nov 11]. Available from: http://www.mhlw.go.jp/stf/seisakunitsuite/bunya/hokabunya/kenkyujigyou/i-kenkyu/
• 32.  Act on Anonymously Processed Medical Information to Contribute to Medical Research and Development [Internet]. [cited 2019 Nov 11]. Available from: http://www.japaneselawtranslation.go.jp/law/detail/?id=3343&vm=04&re=01&new=1
• 33.  Ministry of Education, Culture, Sports, Science and Technology, Ministry of Health, Labour and Welfare & Ministry of Economy T and I. Ethical Guidelines for Human Genome/Gene Analysis Research. 2001.
• 34.   Yamamoto  N,  Fujita  T,  Kawashima  M,  Wittig  J,  Suzuki  M,  Kato  K. The inclusion of genomic data in the 2015 revision of Japan’s Protection of Personal Information Act: protection of wider range of genomic data as our next challenge. J Hum Genet 2018;63:537–538.
• 35.   Minari  J,  Brothers  KB,  Morrison  M. Tensions in ethics and policy created by National Precision Medicine Programs. Hum Genomics 2018;12:22.
• 36.   Corrigan-Curay  J,  Sacks  L,  Woodcock  J. Real-World Evidence and Real-World Data for Evaluating Drug Safety and Effectiveness Opinion. JAMA 2018;320:867–868.
• 37.   Schneeweiss  S,  Glynn  RJ. Real-world data analytics fit for regulatory decision-making. Am J Law Med 2018;44:197–216.
• 38.  21st Century Cures Act - Public Law 114-255. Available from: https://www.govinfo.gov/content/pkg/PLAW-114publ255/pdf/PLAW-114publ255.pdf
• 39.  Real-World Evidence US Food & Drug Administration [Internet]. [cited 2019 Dec 2]. Available from: fda.gov/science-research/science-and-research-special-topics/real-world-evidence
• 40.   Psaty  B,  Breckenridge  A. Mini-Sentinel and Regulatory Science - Big Data Rendered Fit and Functional. N Engl J Med 2014;370:2165–2167.
• 41.   Wedam  S,  Fashoyin-Aje  L,  Bloomquist  E,  Tang  S,  Sridhara  R,  Goldberg  KB, et al. FDA Approval Summary: Palbociclib for Male Patients with Metastatic Breast Cancer. Clin Cancer Res 2019;19:Epub ahead of Print.