Evaluating deep learning approaches for AI-assisted lung ultrasound diagnosis: an international multi-center and multi-scanner study
Keywords:
Lung ultrasound (LUS), Severity score, Artificial intelligence, Deep Learning, Assisted diagnosis, Classification, SegmentationAbstract
Lung ultrasound (LUS) interpretation is often subjective and operator-dependent, motivating the development of automated, artificial intelligence (AI)-based methods. This international, multi-center study evaluated two distinct deep learning approaches for automated LUS severity scoring for pulmonary infections caused by COVID-19: a pre-trained classification model (CM) and a segmentation model based method (SM); assessing performance at video, exam, and prognostic levels. Two datasets were analyzed: one comprising data from multiple scanners and another using data from a single scanner. Results showed that the SM achieved prognostic-level agreement with expert clinicians comparable to that of the CM. Furthermore, at the exam level, over 84% of examinations were classified with acceptable error (≤ 10 score difference) across both models and datasets, reaching both methods an agreement higher than 95% on the dataset acquired by a single scanner. The obtained results demonstrate the potential of AI-assisted LUS for reliable prognostic assessment and highlight that image quality and acquisition technique are key factors in achieving consistent and generalizable model performance, as well as the potential for international clinical translations.
References
1. Mento F, Khan U et al State of the art in lung ultrasound, shifting from qualitative to quantitative analyses. Ultrasound Med Biol 48(12):2398–2416. https://doi.org/10.1016/j.ultrasmedbio.2022.07.007
2. Blazic I, Cogliati C, Flor N, Frija G, Kawooya M, Umbrello M, Ali S, Baranne ML, Cho YJ, Pitcher R, Vollmer I, van Deventer E, Del Rosario Perez M (2023) The use of lung ultrasound in COVID-19. ERJ Open Res 9(1):00196–2022 PMID: 36628270; PMCID: PMC9548241
3. Gutsche H, Lesser TG, Wolfram F, Doenst T (2021) Significance of lung ultrasound in patients with suspected COVID-19 infection at hospital admission. Diagnostics 11(6):921. https://doi.org/10.3390/diagnostics11060921
4. Tung-Chen Y, Martí de Gracia M, Díez-Tascón A, Alonso-González R, Agudo-Fernández S, Parra-Gordo ML, Ossaba-Vélez S, Rodríguez-Fuertes P, Llamas-Fuentes R (2020) Correlation between chest computed tomography and lung ultrasonography in patients with coronavirus disease 2019 (COVID-19). Ultrasound Med Biol 46(11):2918–2926 Epub 2020 Jul 13. PMID: 32771222; PMCID: PMC7357528
5. Soldati G, Smargiassi A, Inchingolo R, Buonsenso D, Perrone T, Briganti DF, Perlini S, Torri E, Mariani A, Mossolani EE, Tursi F, Mento F, Demi L (2020) Proposal for international standardization of the use of lung ultrasound for patients with COVID-19: A simple, quantitative, reproducible method. J Ultrasound Med 39(7):1413–1419 Epub 2020 Apr 13. PMID: 32227492; PMCID: PMC7228287
6. Demi L, Wolfram F, Klersy C, De Silvestri A, Ferretti VV, Muller M, Miller D, Feletti F, Wełnicki M, Buda N, Skoczylas A, Pomiecko A, Damjanovic D, Olszewski R, Kirkpatrick AW, Breitkreutz R, Mathis G, Soldati G, Smargiassi A, Inchingolo R, Perrone T (2023) New international guidelines and consensus on the use of lung ultrasound. J Ultrasound Med 42:309–344. https://doi.org/10.1002/jum.16088
7. Herraiz JL, Freijo C, Camacho J, Muñoz M, González R, Alonso-Roca R, Álvarez-Troncoso J, Beltrán-Romero LM, Bernabeu-Wittel M, Blancas R et al (2023) Inter-Rater variability in the evaluation of lung ultrasound in videos acquired from COVID-19 patients. Appl Sci 13(3):1321
8. Pivetta E, Baldassa F, Masellis S, Bovaro F, Lupia E, Maule MM (2018) Sources of variability in the detection of B-Lines, using lung ultrasound. Ultrasound Med Biol 44(6):1212–1216. https://doi.org/10.1016/j.ultrasmedbio.2018.02.018Epub 2018 Mar 26. PMID: 29598962
9. Lerchbaumer MH, Lauryn JH, Bachmann U et al (2021) Point-of-care lung ultrasound in COVID-19 patients: inter- and intra-observer agreement in a prospective observational study. Sci Rep 11:10678. https://doi.org/10.1038/s41598-021-90153-2
10. Fatima N, Mento F, Zanforlin A, Smargiassi A, Torri E, Perrone T, Demi L (2023) Human-to-AI interrater agreement for lung ultrasound scoring in COVID-19 patients. J Ultrasound Med 42(4):843–851. https://doi.org/10.1002/jum.16052Epub 2022 Jul 7. PMID: 35796343; PMCID: PMC9350219
11. Bajwa J, Munir U, Nori A, Williams B (2021) Artificial intelligence in healthcare: transforming the practice of medicine. Future Healthc J 8(2):e188–e194. https://doi.org/10.7861/fhj.2021-0095PMID: 34286183; PMCID: PMC8285156
12. Zhao L, Lediju Bell MA (2022) A review of deep learning applications in lung ultrasound imaging of COVID-19 patients. BME Front 2022(9780173). https://doi.org/10.34133/2022/9780173
13. Rajpurkar P (2017) CheXNet: Radiologist-Level pneumonia detection on chest X-Rays with deep learning. ArXiv abs/1711, 5225
14. Nam JG, Kim M, Park J, Hwang EJ, Lee JH, Hong JH, Goo JM, Park CM (2021) Development and validation of a deep learning algorithm detecting 10 common abnormalities on chest radiographs. Eur Respir J 57(5):2003061 PMID: 33243843; PMCID: PMC8134811
15. Hannun AY, Rajpurkar P, Haghpanahi M, Tison GH, Bourn C, Turakhia MP, Ng AY (2019) Cardiologist-level arrhythmia detection and classification in ambulatory electrocardiograms using a deep neural network. Nat Med 25(1):65–69. https://doi.org/10.1038/s41591-018-0268-3Epub 2019 Jan 7
16. Howell L, Ingram N, Lapham R, Morrell A, McLaughlan JR (2024) Deep learning for real-time multi-class segmentation of artefacts in lung ultrasound. Ultrasonics 107251. https://doi.org/10.1016/j.ultras.2024.107251
17. Mento F, Perrone T, Fiengo A, Smargiassi A, Inchingolo R, Soldati G, Demi L (2021) Deep learning applied to lung ultrasound videos for scoring COVID-19 patients: a multicenter study. J Acoust Soc Am 149(5):3626. https://doi.org/10.1121/10.0004855
18. Lucassen RT, Jafari MH, Duggan NM, Jowkar N, Mehrtash A, Fischetti C, Bernier D, Prentice K, Duhaime EP, Jin M, Abolmaesumi P, Heslinga FG, Veta M, Duran-Mendicuti MA, Frisken S, Shyn PB, Golby AJ, Boyer E, Wells WM, et al (2023) Deep learning for detection and localization of B-lines in lung ultrasound. IEEE J Biomed Health Inform 27(9): 4352–4361. https://doi.org/10.1109/JBHI.2023.3282596
19. Roy S, Menapace W, Oei S, Luijten B, Fini E, Saltori C, Huijben I, Chennakeshava N, Mento F, Sentelli A, Peschiera E, Trevisan R, Maschietto G, Torri E, Inchingolo R, Smargiassi A, Soldati G, Rota P, Passerini A, van Sloun RJG, Ricci E, Demi L (2020) Deep learning for classification and localization of COVID-19 markers in Point-of-Care lung ultrasound. IEEE Trans Med Imaging 39(8):2676–2687. https://doi.org/10.1109/TMI.2020.2994459Epub 2020 May 14. PMID: 32406829
20. Castro-Sayat M, Colaianni-Alfonso N, Vetrugno L et al (2024) Lung ultrasound score predicts outcomes in patients with acute respiratory failure secondary to COVID-19 treated with non-invasive respiratory support: a prospective cohort study. Ultrasound J 16:20. https://doi.org/10.1186/s13089-024-00365-
21. Dell’Aquila P, Raimondo P, Racanelli V, De Luca P, De Matteis S, Pistone A, Melodia R, Crudele L, Lomazzo D, Solimando AG, Moschetta A, Vacca A, Grasso S, Procacci V, Orso D, Vetrugno L (2022) Integrated lung ultrasound score for early clinical decision-making in patients with COVID-19: results and implications. Ultrasound J 14(1):21. https://doi.org/10.1186/s13089-022-00264-8PMID: 35648278; PMCID: PMC9156837
22. Muñoz M, Rubio A, Cosarinsky G, Cruza JF, Camacho J (2024) Deep Learning-Based algorithms for Real-Time lung ultrasound assisted diagnosis. Appl Sci 14(24):11930. https://doi.org/10.3390/app142411930
23. Khan U, Afrakhteh S, Mento F, Fatima N, De Rosa L, Custode LL, Azam Z, Torri E, Soldati G, Tursi F, Macioce VN, Smargiassi A, Inchingolo R, Perrone T, Iacca G, Demi L (2023) Benchmark methodological approach for the application of artificial intelligence to lung ultrasound data from COVID-19 patients: from frame to prognostic-level. Ultrasonics 132:106994. https://doi.org/10.1016/j.ultras.2023.106994
24. Demi L, Mento F, Di Sabatino A, Fiengo A, Sabatini U, Macioce VN, Robol M, Tursi F, Sofia C, Di Cienzo C, Smargiassi A, Inchingolo R, Perrone T (2022) Lung ultrasound in COVID-19 and Post-COVID-19 patients, an Evidence-Based approach. J Ultrasound Med 41(9):2203–2215 Epub 2021 Dec 3. PMID: 34859905; PMCID: PMC9015439
25. Camacho J, Muñoz M, Genovés V, Herraiz JL, Ortega I, Belarra A, González R, Sánchez D, Giacchetta RC, Trueba- Vicente Á, Tung-Chen Y (2022) Artificial intelligence and democratization of the use of lung ultrasound in COVID-19: on the feasibility of automatic calculation of lung ultrasound score. Int J Transl Med 2:17–25
26. Tung-Chen Y, Ossaba-Vélez S, Acosta Velásquez KS et al (2022) The impact of different lung ultrasound protocols in the assessment of lung lesions in COVID-19 patients: is there an ideal lung ultrasound protocol?? J Ultrasound 25:483–491. https://doi.org/10.1007/s40477-021-00610-x
27. He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition, 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Las Vegas, NV, USA, pp. 770–778. https://doi.org/10.1109/CVPR.2016.90
28. Cohen J (1968) Weighted kappa: nominal scale agreement provision for scaled disagreement or partial credit. Psychol Bull 70(4):213–220. https://doi.org/10.1037/h0026256
29. Landis J, Richard, Koch GG (1977) The measurement of observer agreement for categorical data. Biometrics 33(1):159–174. https://doi.org/10.2307/2529310
30. Perrone T, Soldati G, Padovini L, Fiengo A, Lettieri G, Sabatini U, Gori G, Lepore F, Garolfi M, Palumbo I, Inchingolo R, Smargiassi A, Demi L, Mossolani EE, Tursi F, Klersy C, Di Sabatino A (2021) A new lung ultrasound protocol able to predict worsening in patients affected by severe acute respiratory syndrome coronavirus 2 pneumonia. J Ultrasound Med 40(8):1627–1635. https://doi.org/10.1002/jum.15548Epub 2020 Nov 6. PMID: 33155689
31. Lu W, Zhang S, Chen B, Chen J, Xian J, Lin Y, Shan H, Su ZZ (2020) A clinical study of noninvasive assessment of lung lesions in patients with coronavirus Disease-19 (COVID-19) by bedside ultrasound. Ultraschall Med 41(3):300–307 English. https://doi.org/10.1055/a-1154-8795
32. Mumuni A, Mumuni F (2022) Data augmentation: A comprehensive survey of modern approaches. Array 16:100258. https://doi.org/10.1016/j.array.2022.100258
33. Guan H, Liu M (2022) Domain adaptation for medical image analysis: A survey. IEEE Trans Biomed Eng 69(3):1173–1185 doi: 10.1109/TBME.2021.3117407. Epub 2022 Feb 18. PMID: 34606445; PMCID: PMC9011180
34. Wu Y-H et al (2021) JCS: an explainable COVID-19 diagnosis system by joint classification and segmentation. IEEE Trans Image Process 30:3113–3126. https://doi.org/10.1109/TIP.2021.3058783
Downloads
Published
Issue
Section
License
Copyright (c) 2025 Mario Muñoz, Xi Han, Jorge Camacho, Tiziano Perrone, Andrea Smargiassi, Riccardo Inchingolo, Yale Tung-Chen, Libertario Demi (Author)
This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by-nc/4.0) which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.
Transfer of Copyright and Permission to Reproduce Parts of Published Papers.
Authors retain the copyright for their published work. No formal permission will be required to reproduce parts (tables or illustrations) of published papers, provided the source is quoted appropriately and reproduction has no commercial intent. Reproductions with commercial intent will require written permission and payment of royalties.
