Article ID: CJ-19-0420
Echocardiography has a central role in the diagnosis and management of cardiovascular disease. Precise and reliable echocardiographic assessment is required for clinical decision-making. Even if the development of new technologies (3-dimentional echocardiography, speckle-tracking, semi-automated analysis, etc.), the final decision on analysis is strongly dependent on operator experience. Diagnostic errors are a major unresolved problem. Moreover, not only can cardiologists differ from one another in image interpretation, but also the same observer may come to different findings when a reading is repeated. Daily high workloads in clinical practice may lead to this error, and all cardiologists require precise perception in this field. Artificial intelligence (AI) has the potential to improve analysis and interpretation of medical images to a new stage compared with previous algorithms. From our comprehensive review, we believe AI has the potential to improve accuracy of diagnosis, clinical management, and patient care. Although there are several concerns about the required large dataset and “black box” algorithm, AI can provide satisfactory results in this field. In the future, it will be necessary for cardiologists to adapt their daily practice to incorporate AI in this new stage of echocardiography.
In the modern era, artificial intelligence (AI) is spreading into all parts of daily life. AI is a program that has tasks based on algorithms in an intelligent manner. Machine learning is a subset of AI and focuses on the machine’s ability to receive a set of data and learn for itself. The tasks in machine learning can be classified into supervised and unsupervised learning problems. In the former, the task of assigning data to one of the discrete categories is called classification, whereas the task of fitting the desired output consisting of ≥1 continuous variables is called regression. In the latter, the goal may be to discover some groups categorized with similar variables, or features, called “clustering”. Deep learning is a subset of machine learning that can solve a problem by using multilayered neural networks (Figure 1). Deep learning has led to state-of-the art improvements in word recognition, visual object recognition, object detection, etc.1 Image recognition by machines trained through deep learning in some situations is superior to that of humans. Just a few years ago, we were surprised by a machine learning-based computer program (“AlphaGo”) that defeated the world champion of Go.2 The AI algorithm continues to be enhanced every year. Medical imaging also seems to be changing and undergoing an important revolution because of AI methods such as deep learning based on neural networks.
Artificial intelligence, including machine learning and deep learning and their tasks.
Data are an essential component of AI, and the quality and size of a dataset used to build a model will strongly influence the outcomes. When datasets are biased, the results are unusable in the clinical setting. Preprocessing is also important because echocardiographic data are nonstructural and there are differences in image properties in the dataset. Cardiologists need to make a labeled dataset to develop the model. After development of models, a clinical trial is key to this flow because developed models must be validated in different cohorts. A typical model development process is shown in Figure 2. AI researchers follow this process when developing a new model.
Development process for artificial intelligence models.
Recently, there have been some reports on AI in the medical imaging modalities. For example, calcium scoring in low-dose chest computed tomography (CT), identification of functionally significant stenosis in CT angiography, and diagnosis of chronic myocardial infarction on cine magnetic resonance image (MRI) have been developed.3–5 Compared with CT and MRI, in echocardiography there is an issue of high observer variation in the interpretation of images. Thus, AI might be help to improve observer variation and provide accurate diagnosis in echocardiography. In this review, we focus on the current status and future directions of AI in the field of echocardiography.
Echocardiography has a central role in the diagnosis and management of cardiovascular disease.6 Precise and reliable echocardiographic assessment is required for clinical decision-making.7–10 Even in the development of new technologies (3-dimentional echocardiography, speckle-tracking, semi-automated analysis, etc.), the final analytical decision is strongly dependent on operator experience. For example, left ventricular ejection fraction (LVEF) is subjective, and variability could be influenced by observer experience. Several institutes have several readers with a wide range of experience levels.11,12 Until now, many interventions for reduction of variability in LVEF have been tested to overcome this issue.13,14 Our multicenter group suggested that a simple teaching intervention can reduce the variability in LVEF assessment, especially for readers with limited experience.15 However, there are several limitations, including a lack of ground truth, limited number of sample sizes, etc. Thus, diagnostic errors are a major unresolved problem. Moreover, not only can cardiologists differ from one another in image interpretations, but the same observer may come to different conclusion when a reading is repeated. Daily high workloads in clinical practice may lead to this error, and all cardiologists require precise perception in this field.
AI will likely help to overcome these issues. The AI algorithms might provide an aid to diagnostics with fewer errors and provide hidden features for accurate diagnosis. One landmark echocardiographic paper was recently published.16 The authors used a deep learning model to build a fully automated echocardiogram interpretation program, including view identification, image segmentation, quantification of structure and function, and disease detection. Since then, many cardiologists can see a potential role for AI in the echocardiographic field. Our laboratory also investigated the building of models of automated diagnosis of myocardial infarction using a deep learning algorithm.17 The model showed several new insights and findings in the development of the algorithm. AI has the potential to improve analysis and interpretation of medical images to an advanced stage compared with previous algorithms. Table summarizes the diagnostic ability of current machine-learning models in the field of echocardiography.16,18–25 The remainder of this review focuses on previously published deep learning approaches in echocardiography, view classifications, automated analysis of size and function, diagnosis of cardiovascular diseases, and diastolic dysfunction.
Authors | Year | Target | Models | Training/ validation dataset |
Test dataset |
Accuracy | AUC |
---|---|---|---|---|---|---|---|
Madani et al24 | 2018 | Echocardiography views (Classification) |
Neural network |
200,000 images |
20,000 images |
0.92 | 1.00 |
Zhang et al16 | 2018 | Echocardiography views (Classification) |
Neural network |
Total 14,035 studies |
– | 0.84 | – |
Raghavendra et al25 | 2018 | Wall motion abnormalities (Classification) |
Neural network |
279 images | – | 0.75 | – |
Omar et al18 | 2018 | Wall motion abnormalities (Classification) |
Neural network |
4,392 maps | 61 subjects | 0.95 | – |
Kusunose et al17 | 2019 | Wall motion abnormalities (Classification) |
Neural network (5 types) |
960 images | 240 images+120 images from an independent cohort |
– | 0.97 |
Zhang et al16 | 2018 | LV size and function (Regression*) |
Neural network |
Total 14,035 studies |
– | Median absolute deviations of 15–17% |
– |
Sanchez-Martinez et al19 |
2018 | Heart failure with preserved EF (Clustering) |
Agglomerative hierarchical clustering |
– | – | 0.73 | – |
Tabassian et al20 | 2018 | Heart failure with preserved EF (Clustering and classification) |
KNN and PCA |
– | – | 0.81 | – |
Narula et al21 | 2016 | Myocardial disease (HCM vs. athlete) (Classification) |
Support vector machine |
– | – | – | 0.80 |
Sengupta et al22 | 2016 | Myocardial disease (CP vs. RCM) (Classification) |
Associative memory classifier |
– | – | 0.94 | 0.96 |
Zhang et al16 | 2018 | Myocardial disease (HCM, amyloidosis, PAH) (Classification) |
Neural network |
Total 14,035 studies |
– | – | 0.85–0.93 |
*Left ventricle was segmented by a classification model and then the size and function were evaluated. AUC, area under the curve; CP, constrictive pericarditis; HCM, hypertrophic cardiomyopathy; KNN, k-nearest neighbor; PAH, pulmonary hypertension; PCA, principle component analysis; RCM, restrictive cardiomyopathy.
Echocardiographic images consist of several video clips, still images (M-mode and B-mode) and Doppler recordings because the heart’s structure and function are complex and require many views to diagnose cardiovascular diseases (Figure 3). Because of the nonstructural data in echocardiography, determination of the view is the essential first step in interpreting an echocardiogram. Recent studies applied deep learning with convolutional neural networks for view classification of echocardiograms.24 They trained a convolutional neural network to simultaneously classify 15 standard views (12 video, 3 still), based on labeled still images and videos from 267 transthoracic echocardiograms with over 800,000 images that captured a range of real-world clinical variations. Their model classified among 12 video views with 97.8% overall test accuracy without overfitting. Another group reported that a model for view classification was successfully trained with more layers and a larger number of echocardiography view classes.16 Thus, this method may be reasonable for application to image classification. On the other hand, there are some limitations, including lack of explanation of the learning process and less than perfect classification. The utility is questionable in the current version of models. Moreover, they did not adequately resolve the problem of vendor differences and image qualities. A future enhanced model to classify the correct views would be required.
Variety of echocardiographic images needed to be recognized by artificial intelligence systems.
Quantification of heart size and function is an essential part of echocardiography. Fully automated 3D echocardiographic analysis can obtain quantitative results without any observer interaction (e.g., selection of views, positioning markers and modifying borders). Commercially available software has been tested for accuracy and reproducibility. The algorithms are knowledge-based probabilistic contouring algorithms26 or adaptive analytics algorithms.27 The most frequently used software is the HeartModel algorithm in the Philips EPIQ series (Figure 4). This software shows automated tracings of the left ventricular and left atrial endocardial borders with 3D analysis. There are many studies comparing fully automated methods and either cardiac magnetic resonance or manual echocardiography,28–31 but there are some limitations from the clinical setting viewpoint. One is the dependency on image quality, which has an important role, because results obtained with poor but analyzable image quality provide inaccurate results.32 On the other hand, although measurement accuracy using this analysis still depends on image quality, its degree becomes obviously smaller than with current semi-automated software. The number of datasets for training also affects measurement accuracy. The current adaptive analytics algorithm does not work very well in patients with distorted LV shape, such as LV aneurysms and apical hypertrophic cardiomyopathy because of the limited number of datasets for machine learning.
Representative fully automated analysis software, HeartModel.
These limitations also exist for deep learning in echocardiography. Deep learning algorithms require a high-quality database to provide a sound estimation model with a small sample size. Zhang et al proposed a pipeline based on a deep learning approach for a fully automated analysis of echocardiographic data.16 They proposed to train a U-net deep learning model for automatic segmentation of the LV in the apical 4- and 2-chamber views. For LV segmentation, their model had a Dice score of approximately 85% in the apical 2-chamber view, and approximately 80% in the apical 4-chamber view, and a mean absolute percentage error of approximately 10% for EF from the apical 2-chamber view, and 20% for EF from the apical 4-chamber view. The correlation is good; however, their dataset did not include a wide range of LVEF. In addition, EF values were evaluated after segmentation of the LV, so may include segmentation errors. We believe that a direct evaluation of EF with larger training sets that include extremes of LV range will be needed for training and validation.
One of the most important assessments in echocardiography is evaluating regional wall motion abnormalities (RWMAs) for the management of ischemic coronary artery disease (CAD). Assessment of RWMAs is a Class I recommendation in the guidelines by trained echocardiographic technicians for patients with chest pain in the emergency department.33–35 Conventional assessment of RWMAs, which is based on visual interpretation of endocardial excursion and myocardial thickening, is subjective and experience-dependent.36 A useful method for reducing the misreading of RWMAs is required.37–39 Machine-learning models have been evaluated to identify and quantify RWMAs.18,25 A convolutional neural network provided good models with high sensitivity for diagnosis of CAD. Recently, our laboratory investigated building models of automated diagnosis for myocardial infarction using a deep learning algorithm (Figure 5).17 For detection of the presence of RWMA, the area under the receiver-operating characteristic curve (AUC) by deep learning algorithm was similar to that for a reading by cardiologist/sonographer, and significantly higher than the AUC for resident readers. Interestingly, deep learning had relatively low ratios of misclassification of the right coronary artery, left circumflex coronary artery, and control groups except for the left anterior descending coronary artery (LAD). It seems to reflect the real-world assessment (e.g., overdiagnoses in ischemic groups by human observers or importance of LAD in the clinical setting). The results of a deep learning model in echocardiography might provide new insights in the medical field.
An example of a convolutional neural network model for detection of coronary artery disease. LAD, left anterior descending coronary artery; LCX, left circumflex coronary artery; RCA, right coronary artery.
Several techniques have been applied to identify clinical disease states. In the echocardiographic field, speckle-tracking imaging is widely used in cases of cardiomyopathy. Clinical reports on speckle-tracking imaging show significant differences in regional strain in several cardiomyopathies, even in the absence of ischemia. Knowledge of the characteristic LV strain distribution pattern might facilitate diagnosis of constrictive pericarditis, cardiac amyloidosis, hypertrophic cardiomyopathy, hypertensive heart disease, tachycardia-induced cardiomyopathy, and aortic stenosis (Figure 6).40–46 On the other hand, recent American and European consensus paper describe reginal longitudinal strain assessment by speckle-tracking analysis as still too immature to adopt in the clinical setting.47 The limitation should be overcome in the future. Recently, machine-learning algorithms have revealed clinical disease conditions and new futures. Sengupta et al applied a cognitive machine-learning algorithm to differentiate constrictive pericarditis from restrictive cardiomyopathy with multimodality imaging and pathology.22 The same group showed a machine-learning approach to assessing the potential role diagnosing hypertrophy in athletes and hypertrophic cardiomyopathy.21 Sanchez-Martinez et al19 and Tabassian et al20 showed that machine learning using echocardiographic data, including strain imaging at rest and during exercise, may improve diagnosis and understanding of heart failure with preserved EF. In this field, investigators try to not only assess the accuracy of diagnosis, but also discover new findings in cardiovascular disease. Zhang et al proposed a model based on a deep learning approach for differentiating cardiomyopathy and pulmonary hypertension from the parasternal long-axis views.16 Unlike other machine-learning approaches, the deep learning approach may automatically encode optimal features from data beyond human recognition. Big data have the potential to lead to precise diagnosis and discovering important features from the echocardiographic images. In the future, AI may aid physicians in accurate diagnosis without requiring pathological samples.
Examples of strain distribution in amyloidosis, constrictive pericarditis and hypertrophic cardiomyopathy.
Cardiologists will determine the capability of AI in diagnosis, and they will be responsible for the final decisions. Thus, cardiologists will be required to have the capacity e to manage AI and advanced knowledge. Some recent studies have been concerned about adversarial examples in the medical imaging field.48 Adversarial examples are inputs to learning models that an attacker has intentionally designed to cause the model to make a mistake; they are like optical illusions for machines. In echocardiography, data are just pixel images, not structured data. Echocardiographic imaging systems may be vulnerable to adversarial attacks. For example, insurance companies will use a deep learning system that receives images as part of a claim to verify that heart surgery would be necessary in the future. An adversarial example may be used to deceive the insurance company’s system. In these cases, cardiologists should have adequate and solid knowledge in this field. The era of AI is almost here.
From our comprehensive review, we believe AI has the potential to improve accuracy of diagnosis, clinical management, and patient care. Although there are several concerns about the required large dataset and “black box” algorithm, AI seems able to provide satisfactory results in this field. In the future, it will be necessary for cardiologists to incorporate this new horizon of AI in echocardiography into their daily practice.
This work was partially supported by JSPS Kakenhi Grants (Number 17K09506 to K. Kusunose, and 19H03654 to M. Sata), the Takeda Science Foundation (to M. Sata), and the Vehicle Racing Commemorative Foundation (to M. Sata).
The authors declare no conflicts of interest.
The authors acknowledge Kathryn Brock, BA, for revising the manuscript.