Audit-Based Quality Validation of the Japan Cardiovascular Surgery Database

Abstract

Background: The Japan Cardiovascular Surgery Database (JCVSD) is a nationwide registry of patients undergoing cardiovascular surgery in Japan. To investigate and improve data quality, we have been conducting on-site institutional audits since 2004. This study aimed to investigate the accuracy of the registered data by comparing it to site visit data.

Methods and Results: The subjects of this study were the 95 facilities at which a site visit was conducted. The case registration accuracy was 98.74%. Furthermore, we confirmed high data input accuracy of >90% for almost all fields. Approximately 99% of cases had been correctly entered for diabetes, aortic stenosis, and mortality. We also discovered which fields were more likely to be incorrectly captured and the causes thereof, as well as problems regarding some definitions and the input system itself.

Conclusions: We were able to confirm high registration accuracy in the JCVSD. Appropriately resourced, focused site visits as part of a national audit are capable of accurate data collection on which continual nationwide quality control can be based. Continued work and development to further improve the quality of the database are mandatory to maintain a high standard of cardiovascular surgery in Japan.

In recent years, numerous medical databases have been constructed in a wide range of fields with the goal of improving the quality of medical care and welfare in each field. In the field of cardiovascular surgery, the Japan Cardiovascular Surgery Database (JCVSD) has been constructed for this purpose.¹ As of April 2018, the database comprised 584 facilities and 560,788 cases, and, since 2011, all patients undergoing cardiovascular surgery have been registered from all facilities around the nation.

The input fields were created based on the Society of Thoracic Surgeons (STS) National Database, and the number of fields has increased to approximately 500. Clinical research projects using these national data have been conducted,^2–7 and the results of the analysis of national data have been disseminated worldwide. A risk model was constructed based on the JCVSD data, from which JapanSCORE, a risk analysis model in Japan, was developed, which has proven useful since its deployment in 2007.^8–10 In addition, the number of all cardiothoracic surgery cases in Japan has been reported annually by the Committee for Scientific Affairs of the Japanese Association for Thoracic Surgery.^11,12 Furthermore, biannual JCVSD update reports include not only the number of cases, but also statistical data regarding the background information of each patient, such as age and preoperative risk factors.^13–16

In medical databases, data are usually collected and entered at the facility level by the data manager at each facility. At the input stage, invalid data entry may occur due to misinterpretation of the contents of the medical records, input errors, and a misunderstanding of definitions. Therefore, validation of input data is essential. One of the most effective methods to evaluate accuracy is by directly comparing input data to the case data of the institution. We implemented the site visit procedure in an effort to improve database reliability and data quality. This study evaluated the accuracy of the input data compared with the results of the site visits.

Methods

Because this work focuses on the structure of the database itself and does not involve patient information, the need for Institutional Review Board (IRB) approval was waived. Of course, each hospital has obtained IRB approval from the corresponding institution to contribute to the database.

The Site Visit Working Group (SV-WG) was established in 2006 by cardiovascular surgeons from multiple centers across the country to perform regular audits. This committee determines the implementation methods and policies of the site visit. The facilities targeted for site visits were selected from among facilities with a long history of using the JCVSD. These facilities were then contacted and asked for their consent to a site visit. In addition, site visits were preferentially implemented at facilities that have applied for projects using this national database for analysis. The auditors were 4–6 cardiovascular surgeons (mainly SV-WG members) who have no interest in the target facilities.

There are 4 steps in site visit implementation:

1. Data cleaning: organizing and validating the content of patient data input in the facility before the visit and identifying and listing any issues related to the input characteristics of the facility

2. On-site inspection: validating the number of registered cases using the operating ledger and input data from that particular year, validating the medical records and input data of cases to be surveyed, and conducting an audit summary meeting at the end of the site visit with the data manager and responsible person for the facility

3. Delivery of reports and aggregation of results

4. Reporting the results to all facilities nationwide at the biannual data manager meeting.

For the input data, cases were classified according to whether the patient was deceased or living. The total number of surgical deaths recorded from the facilities at the time of auditing was 3,821. From these, we selected 1,091 (28.6%) recent cases and confirmed all fields. Regarding living patients, 1,055 (8.2%) cases from the most recent year were randomly selected from a total of 12,948 records from the facilities. We then checked important fields affecting the risk model. In addition, in the case of living patients, all cases with serious problems identified during data cleaning were included in the inspection.

From July 2004 to April 2018, site visits were conducted at 100 (17.1%) of the 584 participating facilities. These 584 facilities account for almost all centers performing cardiovascular surgery in Japan. The number of targeted registered cases at the time of implementation was 82,462 of a total of 560,788 cases (14.7%). In this study, the initial 5 facilities, whose input status was not evaluated quantitatively in the report, were excluded and 95 facilities were targeted. The number of cases to be validated was 79,695. Validation of the operating ledger, which is the list of surgical cases at each facility, was performed on 13,606 cases, and validation of the medical records was performed on 2,553 cases (Table 1). In addition, site visits were performed by a total of 349 cardiovascular surgeons.

Table 1. Subjects of the Site Visit Study

	Number	Unit
Institutions	95	Institutions
Study cases	79,695	Cases
Operating ledger confirmation	13,606	Cases
Confirmation of medical records	2,553	Cases

The accuracy of the number of registrations was validated for the operating ledger. The ledger was also checked for age and sex, and facilities that did not list this information were excluded. Therefore, we targeted 88 facilities and 12,110 cases. Validation of important fields affecting the risk model was conducted using medical records for 95 facilities and 2,541 cases (Table 2).

Table 2. Number of Subjects

	Subject	No. institutions	No. cases
No. cases	Operating ledger	95	13,606
Age and sex	Operating ledger	88	12,110
Preoperative fields influencing the risk model, operative method, and outcome	Medical records	95	2,541

Fields that are important for creating a risk model were selected for the validation of input accuracy. These included age, sex, and 13 fields covering preoperative risk, such as diabetes and renal dysfunction (Table 3A). An operative method field, 5 fields for postoperative information such as reoperation and dialysis, and 2 fields for mortality were included as outcome fields (Table 3B).

Table 3. Confirmed Fields

(A) Fields affecting the risk model

Age

Sex

Preoperative diabetes

Preoperative renal dysfunction

Preoperative creatinine

Preoperative COPD

Preoperative extracardiac arteriopathy

History of cardiovascular surgery

Heart failure

Cardiogenic shock

NYHA classification

Inotropic agents

LV function

Aortic stenosis

Urgency

(B) Outcome and operative method fields

Operative method

Complications in hospital

Reoperation for any reason

Stroke

Dialysis

Mediastinal infection

Prolonged ventilation

Mortality

Status at 30 days

Status at discharge

COPD, chronic obstructive pulmonary disease; LV, left ventricular; NYHA, New York Heart Association.

Results

Number of Registrations

The accuracy of the number of registrations was validated using the operating ledger. There were 332 unregistered cases, and 172 cases were ineligible for registration, including duplicate and multiple registration cases, accounting for 2.44% and 1.26% of the total, respectively (Table 4). Regarding the number of facilities with incorrect data input, 45 facilities (47.4%) had unregistered cases, and 47 facilities (49.5%) had cases that were ineligible for registration.

Table 4. Status of Registrations

Registration status	n (%)
Missing	332 (2.44)
Ineligible	172 (1.26)

There were 95 target facilities and 13,606 cases. “Missing” refers to unregistered cases; “ineligible” refers to cases that were ineligible for registration, including cases with duplicate and multiple registrations.

Accuracy of Input Data

For age and sex, 98.8% and 99.2% of inputs, respectively, were correctly entered (Table 5). Of the 88 facilities targeted for validation of accuracy, 53 (60.2%) and 37 (42.0%) registered incorrect data for age and sex, respectively, although only a small number of errors occurred in each facility. Table 6 shows the result of confirming the important fields affecting the risk model, surgical procedures, and outcomes in the medical records. An input accuracy of >90% was found in almost all fields examined. For the preoperative fields, the New York Heart Association (NYHA) classification field yielded a relatively low accuracy rate (88%). In addition, accuracy was relatively low for preoperative renal dysfunction (91.6%) and extracardiac arteriopathy lesions (90.2%). In contrast, diabetes and aortic valve stenosis inputs were found to be very accurate, with 98.7% and 98.5% accuracy, respectively. The surgical and postoperative fields also showed an accuracy of ≥93%; in particular, mortality showed a very high accuracy of 98.4% at 30 days and 99.6% at discharge.

Table 5. Accuracy of Age and Sex Data Registration

Element	n (%)
Age	11,965 (98.8)
Sex	12,014 (99.2)

There were 88 target facilities and 12,110 cases.

Table 6. Accuracy of Registration for Preoperative Fields Influencing the Risk Model and Operation and Outcome Fields

	n (%)
(A) Preoperative fields influencing the risk model
Preoperative diabetes	2,509 (98.7)
Preoperative renal dysfunction	2,328 (91.6)
Preoperative creatinine	2,435 (95.8)
Preoperative COPD	2,429 (95.6)
Preoperative extracardiac arteriopathy	2,292 (90.2)
History of cardiovascular surgery	2,466 (97.0)
Heart failure	2,399 (94.4)
Cardiogenic shock	2,454 (96.6)
NYHA classification	2,235 (88.0)
Inotropic agents	2,428 (95.6)
LV function	2,464 (97.0)
Aortic stenosis	2,502 (98.5)
Urgency	2,401 (94.5)
(B) Operation and outcome fields
Operative method	2,382 (93.7)
Complications in hospital
Reoperation for any reason	2,375 (93.5)
Stroke	2,426 (95.5)
Dialysis	2,416 (95.1)
Mediastinal infection	2,520 (99.2)
Prolonged ventilation	2,417 (95.1)
Mortality
Status at 30 days	2,501 (98.4)
Status at discharge	2,532 (99.6)

There were 95 target facilities and 2,541 cases. COPD, chronic obstructive pulmonary disease; LV, left ventricular; NYHA, New York Heart Association.

Discussion

Regarding accuracy, there were 2 primary areas of interest: registered case numbers and data input. For case numbers, very few cases were ineligible for registration or were unregistered. For data input, accuracy exceeded 90% in nearly all 23 fields, including age and sex.

Differences in the Accuracy of Each Field

Input data for preoperative renal dysfunction, extracardiac arteriopathy lesions, and NYHA classification tended to be less accurate than for other fields. It is possible that the definition of each of these fields was not clearly understood, and, in many cases, the objective findings were not sufficiently described in the medical records. That is, incorrect inputs tended to be noticeable in fields where it was difficult for the data manager to understand the medical record. Consequently, we believe that the evaluation of the patient’s condition should be performed by individuals who can accurately understand it, such as the attending physician and/or the nurse in charge. Conversely, we think that fields with clear definitions, such as diabetes, aortic valve stenosis, and discharge conditions, had very high accuracy because they were commonly understood at each facility.

Improvement of the Input System and Accuracy

Although problems with the input system have been pointed out, we feel it is possible to eliminate errors at the input stage by building checks and warnings into the system. Therefore, based on the indications from site visits, an input check system was built into the input software for fields that tend to be excluded at the input stage and fields that are commonly captured incorrectly. Taking the aforementioned items with a high frequency of erroneous input as an example, in the new input check system preoperative renal dysfunction is defined by the creatinine level and the presence of dialysis, and a warning message is displayed when there is a discrepancy in those inputs. Similarly, for the extracardiac vascular lesions and NYHA classification fields, if there is a discrepancy with the data of specific vascular lesions and heart disease, the input operation is rejected by the system. This system was implemented in 2018, and a significant reduction in erroneous input is expected. In addition, we are currently building a more precise check system for the input software.

Auditing in Other Registries

The STS started an audit in 2006 with the aim of confirming completion rate and accuracy; these activities were being conducted by external independent organizations. The matching rate of data elements for coronary artery bypass graft (CABG) and valve surgery has been confirmed, and is reported to be in the range of 94.5–97.2% by annual aggregation.^17–19 Moreover, in China, audits targeting submitted in-hospital and follow-up data on CABG and valve operations from large cardiac surgical care centers were conducted; the completion rate was reported to be 97.6% and registration accuracy was 95.1%.²⁰ In congenital heart disease, on-site audits (accompanied by an experienced congenital cardiac surgeon) by the STS, the European Association for Cardio-Thoracic Surgery, and the United Kingdom Central Cardiac Audit Database have been reported on procedures performed by congenital cardiac surgeons.²¹ Initially, problems with the input data, such as prognosis, were identified, but in subsequent reports it was shown that the completion rates of general data and mortality were 97.6% and 100%, respectively, with an accuracy of data input of 97.4% and 99.3%, respectively.^21,22 The congenital heart disease section of the JCVSD also reported the accuracy of the registry, with a high concordance rate of 98–100% for age, sex, and operation time, and 100% accuracy for mortality.²³ In addition, the European Society of Thoracic Surgeons conducted audits of facilities by independent external organizations.²⁴ In other cardiovascular areas, the Transcatheter Valve Therapy Registry was established by the STS and the American College of Cardiology and was audited using electronic media; an overall agreement rate of 83.4% for audited procedures that occurred between October 1, 2013 and June 30, 2014 was reported.²⁵

Globally, audits are considered essential to maintaining and improving the quality of databases for the accurate collection of surgical case data and risk model construction. In the present study, a very high input accuracy was confirmed in the JCVSD. The high accuracy rate of data registration and input was attributed to the fact that audits were conducted by the SV-WG, an internal organization that is not only familiar with the items and inputs of the JCVSD, but also continuously engaged in JCVSD activities, with the results of the audit were immediately reflected in the data. In contrast, audits also raise problems concerning input fields and input systems. It is necessary to address these issues as well as aim to maintain and further improve the quality of the data.

Study Limitations

There were 3 types of confirmed cases in site visits: those of deceased patients, those of living patients for which data cleaning revealed data input errors, and those of randomly selected living patients. There tended to be a greater frequency of errors in the case of data for deceased patients and cases of data cleaning than for cases of randomly selected living patients; therefore, in the aggregation as a whole, the erroneous input rate may have been overestimated.

Conclusions

The very high accuracy of the JCVSD input was confirmed by site visits. In addition, potential improvements were identified regarding input fields and input systems. Site visits will continue to be conducted at facilities throughout the country to check the accuracy of input data and identify additional issues that may arise. Based on our findings, we have constructed an input system that will reduce incorrect data entry and, as a result, improve the overall quality of the data. The ultimate goal will be to build the most accurate and reliable national cardiovascular surgery database, contributing to the progression and promotion of cardiovascular surgery worldwide.

Acknowledgments

The authors are deeply grateful to Masataka Mitsuno, Yoshito Kawachi, and Masanao Nakai, who were engaged in site visit activity and examined the accuracy of the database, as well as the quality of the data itself. The authors also express their gratitude to Katsuhiko Oda, Hiroyuki Yamamoto, Norifumi Otani, Akihito Mikamo, Takashi Yamauchi, Noritsugu Shiono, Yoshinori Ohtsu, Wataru Tatsuishi, and Shinya Takase (in order of membership), who have been working as site visit reviewers and data evaluators. The authors thank Professors Ichiro Sakuma and Minoru Ono for their continuous support and Yukari Kato, Makoto Okada, Asako Oi, and the staff in charge of the JCVSD who provided administrative support to help activities run smoothly. Finally, the authors express their deepest appreciation to the data managers and staff in each facility who were constantly making efforts to enter data daily and maintain the accuracy of the database.

Sources of Funding

This activity was funded by the JCVSD organization.

Disclosures

The authors declare no conflicts of interest.

IRB Information

Because this work focused only on the structure of the database and did not involve patient information, the JCVSD organization determined that the study was classified as “Category A” per the ethical guidelines and the IRB process was waived.

Data Availability

The deidentified participant data will not be shared.

References

• 1.   Takamoto S, Motomura N, Miyata H, Tsukihara H. Current status of cardiovascular surgery in Japan, 2013 and 2014: A report based on the Japan Cardiovascular Surgery Database (JCVSD). 1: Mission and history of JCVSD. Gen Thorac Cardiovasc Surg 2018; 66: 1–3.
• 2.   Motomura N, Miyata H, Tsukihara H, Okada M, Takamoto S, Japan Cardiovascular Surgery Database Organization. First report on 30-day and operative mortality in risk model of isolated coronary artery bypass grafting in Japan. Ann Thorac Surg 2008; 86: 1866–1872.
• 3.   Motomura N, Miyata H, Tsukihara H, Takamoto S, Japan Cardiovascular Surgery Database Organization. Risk model of thoracic aortic surgery in 4707 cases from a nationwide single-race population through a web-based data entry system: The first report of 30-day and 30-day operative outcome risk models for thoracic aortic surgery. Circulation 2008; 118(Suppl): S153–S159.
• 4.   Motomura N, Miyata H, Tsukihara H, Takamoto S, Japan Cardiovascular Surgery Database Organization. Risk model of valve surgery in Japan using the Japan Adult Cardiovascular Surgery Database. J Heart Valve Dis 2010; 19: 684–691.
• 5.   Saito A, Motomura N, Miyata H, Takamoto S, Kyo S, Ono M, et al. Age-specific risk stratification in 13 488 isolated coronary artery bypass grafting procedures. Interact Cardiovasc Thorac Surg 2011; 12: 575–580.
• 6.   Saito A, Kumamaru H, Ono M, Miyata H, Motomura N. Propensity-matched analysis of a side-clamp versus an anastomosis assist device in cases of isolated coronary artery bypass grafting. Eur J Cardiothorac Surg 2018; 54: 889–895.
• 7.   Yamauchi T, Miyata H, Sakaguchi T, Miyagawa S, Yoshikawa Y, Takeda K, et al. Coronary artery bypass grafting in hemodialysis-dependent patients: Analysis of Japan Adult Cardiovascular Surgery Database. Circ J 2012; 76: 1115–1120.
• 8.   Miyata H, Motomura N, Tsukihara H, Takamoto S, Japan Cardiovascular Surgery Database. Risk models including high-risk cardiovascular procedures: Clinical predictors of mortality and morbidity. Eur J Cardiothorac Surg 2011; 39: 667–674.
• 9.   Miyata H, Tomotaki A, Motomura N, Takamoto S. Operative mortality and complication risk model for all major cardiovascular operations in Japan. Ann Thorac Surg 2015; 99: 130–139.
• 10.   Umehara N, Saito S, Tsukui H, Yamazaki K. Usefulness of JapanSCORE: Comparative study of the usefulness of the JapanSCORE and the logistic EuroSCORE. Jpn J Cardiovasc Surg 2013; 42: 94–102.
• 11.   Committee for Scientific Affairs, The Japanese Association for Thoracic Surgery; Shimizu H, Endo S, Natsugoe S, Doki Y, Hirata Y, Kobayashi J, et al. Thoracic and cardiovascular surgery in Japan during 2016: Annual report by The Japanese Association for Thoracic Surgery. Gen Thorac Cardiovasc Surg 2019; 67: 377–411.
• 12.   Committee for Scientific Affairs, The Japanese Association for Thoracic Surgery; Shimizu H, Okada M, Tangoku A, Doki Y, Endo S, Fukuda H, et al. Thoracic and cardiovascular surgeries in Japan during 2017: Annual report by the Japanese Association for Thoracic Surgery. Gen Thorac Cardiovasc Surg 2020; 68: 414–449.
• 13.   Hirata Y, Hirahara N, Murakami A, Motomura N, Miyata H, Takamoto S. Current status of cardiovascular surgery in Japan 2013 and 2014: A report based on the Japan Cardiovascular Surgery Database. 2: Congenital heart surgery. Gen Thorac Cardiovasc Surg 2018; 66: 4–7.
• 14.   Saito A, Hirahara N, Motomura N, Miyata H, Takamoto S. Current status of cardiovascular surgery in Japan, 2013 and 2014: A report based on the Japan Cardiovascular Surgery Database 3. Coronary artery bypass surgery. Gen Thorac Cardiovasc Surg 2018; 66: 8–12.
• 15.   Nakano K, Hirahara N, Motomura N, Miyata H, Takamoto S. Current status of cardiovascular surgery in Japan, 2013 and 2014: A report based on the Japan Cardiovascular Surgery Database. 4: Valvular heart surgery. Gen Thorac Cardiovasc Surg 2018; 66: 13–18.
• 16.   Shimizu H, Hirahara N, Motomura, Miyata H, Takamoto S. Current status of cardiovascular surgery in Japan, 2013 and 2014: A report based on the Japan Cardiovascular Surgery Database 5. Thoracic aortic surgery. Gen Thorac Cardiovasc Surg 2017; 65: 671–678.
• 17.   Winkley Shroyer AL, Bakaeen F, Shahian DM, Carr BM, Prager RL, Jacobs JP, et al. The Society of Thoracic Surgeons Adult Cardiac Surgery Database: The driving force for improvement in cardiac surgery. Semin Thorac Cardiovasc Surg 2015; 27: 144–151.
• 18.   D’Agostino RS, Jacobs JP, Badhwar V, Paone G, Rankin JS, Han JM, et al. The Society of Thoracic Surgeons Adult Cardiac Surgery Database: 2017 Update on outcomes and quality. Ann Thorac Surg 2017; 103: 18–24.
• 19.   Jacobs JP, Shahian DM, He X, O’Brien SM, Badhwar V, Cleveland JC Jr, et al. Penetration, completeness, and representativeness of The Society of Thoracic Surgeons Adult Cardiac Surgery Database. Ann Thorac Surg 2016; 101: 33–41.
• 20.   Rao C, Zhang H, Gao H, Zhao Y, Yuan X, Hua K, et al. The Chinese Cardiac Surgery Registry: Design and data audit. Ann Thorac Surg 2016; 101: 1514–1521.
• 21.   Clarke DR, Breen LS, Jacobs ML, Franklin RCG, Tobota Z, Maruszewski B, et al. Verification of data in congenital cardiac surgery. Cardiol Young 2008; 18(Suppl 2): 177–187.
• 22.   Overman DM, Jacobs ML, O’Brien JE Jr, Kumar SM, Mayer JE Jr, Ebel A, et al. Ten years of data verification: The Society of Thoracic Surgeons Congenital Heart Surgery Database audits. World J Pediatr Congenit Heart Surg 2019; 10: 454–463.
• 23.   Takahashi A, Kumamaru H, Tomotaki A, Matsumura G, Fukuchi E, Hirata Y, et al. Verification of data accuracy in Japan Congenital Cardiovascular Surgery Database including its postprocedural complication reports. World J Pediatr Congenit Heart Surg 2018; 9: 150–156.
• 24.   Falcoz PE, Brunelli A. The European General Thoracic Surgery Database project. J Thorac Dis 2014; 6(Suppl 2): S272–S275.
• 25.   Grover FL, Vemulapalli S, Carroll JD, Edwards FH, Mack MJ, Thourani VH, et al. 2016 Annual report of The Society of Thoracic Surgeons/American College of Cardiology Transcatheter Valve Therapy Registry. J Am Coll Cardiol 2017; 69: 1215–1230.