Cedars-Sinai and Smidt Heart Institute investigators developed a novel foundation model that integrates computer vision interpretation of echocardiogram images with natural language processing to augment cardiologists’ interpretation of echocardiograms. Image by Getty.
May 2, 2024 — Artificial intelligence experts at Cedars-Sinai and the Smidt Heart Institute created a dataset with more than 1 million echocardiograms, or cardiac ultrasound videos, and their corresponding clinical interpretations. Using this database, they created EchoCLIP, a powerful machine learning algorithm that can “interpret” echocardiogram images and assess key findings.
The design and evaluation of EchoCLIP, described in a manuscript published in the peer-reviewed journal Nature Medicine, suggest that an EchoCLIP interpretation of a patient’s echocardiogram provides clinician-level evaluations of heart function, assessment of past surgeries and devices, and may assist clinicians in identifying patients in need of treatment. The EchoCLIP foundation model also can identify the same patient across multiple videos, studies and timepoints as well as recognize clinically important changes in a patient’s heart.
“To our knowledge, this is the largest model trained on echocardiography images,” said corresponding author David Ouyang, MD, a faculty member in the Department of Cardiology in the Smidt Heart Institute and in the Division of Artificial Intelligence in Medicine. “Many previous AI models for echocardiograms are only trained on tens of thousands of examples. In contrast, EchoCLIP’s uniquely strong performance in image interpretation is a result of its training on almost tenfold more data than existing models.”
“Our results suggest that large datasets of medical imaging and expert-adjudicated interpretations can serve as the basis for training medical foundation models, which are a form of generative artificial intelligence,” Ouyang said. He said this advanced foundation model can soon help cardiologists in the assessment of echocardiograms by generating preliminary assessments of cardiac measurements, identify changes that happen over time, and common disease states.
The team of investigators built a dataset of 1,032,975 cardiac ultrasound videos and corresponding expert interpretations to develop EchoCLIP. Key takeaways from the study include:
- EchoCLIP displayed strong performance when assessing cardiac function using heart images.
- The foundation model could identify implanted intracardiac devices like a pacemaker, implanted mitral valve repairs and aortic valves from the echocardiogram images.
- EchoCLIP accurately identified unique patients across studies, identified clinically important changes such as having undergone heart surgery, and enabled the development of a preliminary text interpretation of echocardiogram images.
“Foundation models are one of the newest areas within generative AI, but most models do not have enough medical data to be useful in the healthcare arena,” said Christine M. Albert, MD, MPH, chair of the Department of Cardiology in the Smidt Heart Institute and the Lee and Harold Kapelovitz Distinguished Chair in Cardiology.
Albert, who was not involved in the Nature Medicine study, said, “This novel foundation model integrates computer vision interpretation of echocardiogram images with natural language processing to augment cardiologists’ interpretation of echocardiograms.”
For more information: www.cedars-sinai.org
October 28, 2025 
