Fully automatic segmentation of arbitrary anatomical structures from 3D medical image data is a challenging, yet unsolved problem. Though fully automatic segmentation is essential for further clinical analysis, complexity of anatomical structures across population makes a generalized segmentation scheme extremely challenging. Moreover, specific challenges of different imaging modalities have so far hindered the possibility of a general purpose fully automatic 3D segmentation framework. Statistical 3D shape models have proven to be valuable shape priors that are to be deformed within their range of normal variation in shape to match the respective image information. Within the project, we are aiming to combine Machine Learning along with the statistical shape priors for getting a step closer to a general 3D image segmentation approach. In particular, Machine Learning techniques for image matching based on intensity will be developed in order to improve both the model building as well as the segmentation process.

 

Modal General 3D Segmentation Teaser

 

Image-based Cost Functions

Principal Component Analysis (PCA) on local intensity profiles has not proven to beneficially act as a robust cost function. Random Forest Regression Voting (RFRV), though a powerful method for 2D image data, turned out to be impractical for 3D data, due to huge memory consumption and computational time. Dictionary Learning (DL) does not require any heuristics and is general enough to be applied across anatomies and modalities. DL operations are matrix operations, thus being efficiently evaluated.

 

Joint Dictionary Learning

Given 3D image data and accordingly segmented anatomical structures of interest, rotational invariant histograms of oriented gradients (HoG) are sampled at the structures’ boundaries. These feature samples are used as input for learning a dictionary.

 
JDL Method

A second dictionary is learnt for background image information. A combined dictionary of foreground and background features has been established, acting as a cost function for image segmentation.

Cost Function generated by JDL

Cost Function for a test patch: Sum of residuals from representations by the two dictionaries.

 

Clustered Appearance Models

Local appearance patterns of anatomical structures in medical image data play an important role in determining the location, yet the variation of these patterns across localities are challenging for traditional appearance models. Dictionary Learning, even though widely used as an appearance model, lacks a systematic approach for comparison between dictionaries for subsequent clustering. A dictionary comparison framework is proposed for generating clustered appearance models useful for fully automatic segmentation.

Regional Appearance Dictionaries

We consider available dictionary comparison metrics and propose two new metrics for comparison. Three novel meta comparison schemes are also developed for systematic comparison of the metrics in appearance dictionary context. Furthermore, a novel non-overlapping dictionary clustering approach is developed for clustered appearance modeling on anatomical structures.

Quality of clustering

Initial Seeds (upper row) and the resulting clusters (bottom row) by our dictionary clustering algorithm. Notice the similarity of clusters for di fferent number of initial seeds: (a) 50, (b) 150, (c) 250, (d) 350 and (e) 450.

Publications

2021
AutoImplant 2020 - First MICCAI Challenge on Automatic Cranial Implant Design IEEE Transactions on Medical Imaging, 40(9), pp. 2329-2342, 2021 Jianning Li, Pedro Pimentel, Angelika Szengel, Moritz Ehlke, Hans Lamecker, Stefan Zachow, Laura Estacio, Christian Doenitz, Heiko Ramm, Haochen Shi, Xiaojun Chen, Franco Matzkin, Virginia Newcombe, Enzo Ferrante, Yuan Jin, David G. Ellis, Michele R. Aizenberg, Oldrich Kodym, Michal Spanel, Adam Herout, James G. Mainprize, Zachary Fishman, Michael R. Hardisty, Amirhossein Bayat, Suprosanna Shit, Bomin Wang, Zhi Liu, Matthias Eder, Antonio Pepe, Christina Gsaxner, Victor Alves, Ulrike Zefferer, Cord von Campe, Karin Pistracher, Ute Schäfer, Dieter Schmalstieg, Bjoern H. Menze, Ben Glocker, Jan Egger BibTeX
DOI
Machine Learning Approaches for Enhanced, Shape Model Based 3D Image Segmentation
2017
Biharmonic Density Estimate - a scale space descriptor for 3D deformable surfaces Pattern Analysis and Application, pp. 1-13, 2017 Anirban Mukhopadhyay, Suchendra Bhandarkar BibTeX
DOI
Machine Learning Approaches for Enhanced, Shape Model Based 3D Image Segmentation
Statistical shape modeling of the left ventricle: myocardial infarct classification challenge IEEE Journal of Biomedical and Health Informatics, 2017 Avan Suinesiaputra, Pierre Albin, Xenia Alba, Martino Alessandrini, Jack Allen, Wenjia Bai, Serkan Cimen, Peter Claes, Brett Cowan, Jan D'hooge, Nicolas Duchateau, Jan Ehrhardt, Alejandro Frangi, Ali Gooya, Vicente Grau, Karim Lekadir, Allen Lu, Anirban Mukhopadhyay, Ilkay Oksuz, Nripesh Parajuli, Xavier Pennec, Marco Pereanez, Catarina Pinto, Paolo Piras, Marc-Michael Rohe, Daniel Rueckert, Dennis Saring, Maxime Sermesant, Kaleem Siddiqi, Mahdi Tabassian, Lusiano Teresi, Sotirios Tsaftaris, Matthias Wilms, Alistair Young, Xingyu Zhang, Pau Medrano-Gracia BibTeX
DOI
Machine Learning Approaches for Enhanced, Shape Model Based 3D Image Segmentation
Validation of three-dimensional models of the distal femur created from surgical navigation point cloud data for intraoperative and postoperative analysis of total knee arthroplasty International Journal of Computer Assisted Radiology and Surgery, 12(12), pp. 2097-2105, 2017 David Wilson, Carolyn Anglin, Felix Ambellan, Carl Martin Grewe, Alexander Tack, Hans Lamecker, Michael Dunbar, Stefan Zachow PDF
BibTeX
DOI
Machine Learning Approaches for Enhanced, Shape Model Based 3D Image Segmentation
2016
Detection and Characterization of Intrinsic Symmetry of 3D Shapes Proceedings of IEEE International Conference on Pattern Recognition, 2016 Anirban Mukhopadhyay, Fatih Porikli, Suchendra Bhandarkar BibTeX
Machine Learning Approaches for Enhanced, Shape Model Based 3D Image Segmentation
Joint Geometric Graph Embedding for Partial Shape Matching in Images IEEE Winter Conference on Applications of Computer Vision, pp. 1-9, IEEE, IEEE Winter Conference on Applications of Computer Vision (WACV), 2016 Anirban Mukhopadhyay, Arun Kumar, Suchendra Bhandarkar BibTeX
Machine Learning Approaches for Enhanced, Shape Model Based 3D Image Segmentation
Robust and Accurate Appearance Models Based on Joint Dictionary Learning Data from the Osteoarthritis Initiative Lecture Notes in Computer Science, Patch-Based Techniques in Medical Imaging. Patch-MI 2016, pp. 25-33, Vol.9993, 2016 Anirban Mukhopadhyay, Oscar Morillo, Stefan Zachow, Hans Lamecker BibTeX
DOI
Machine Learning Approaches for Enhanced, Shape Model Based 3D Image Segmentation
Total Variation Random Forest: Fully automatic MRI segmentation in congenital heart disease RAMBO 2016, HVSMR 2016: Reconstruction, Segmentation, and Analysis of Medical Images, pp. 165-171, Vol.LNCS 10129, 2016 Anirban Mukhopadhyay BibTeX
DOI
Machine Learning Approaches for Enhanced, Shape Model Based 3D Image Segmentation
2015
Data-Driven Feature Learning for Myocardial Segmentation of CP-BOLD MRI Functional Imaging and Modeling of the Heart, pp. 189-197, Vol.9126, Lecture Notes in Computer Science, 2015 Anirban Mukhopadhyay, Ilkay Oksuz, Marco Bevilacqua, Rohan Dharmakumar, Sotirios Tsaftaris BibTeX
DOI
Machine Learning Approaches for Enhanced, Shape Model Based 3D Image Segmentation
Dictionary Learning Based Image Descriptor for Myocardial Registration of CP-BOLD MR Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2015, pp. 205-213, Vol.9350, Lecture Notes in Computer Science, 2015 Ilkay Oksuz, Anirban Mukhopadhyay, Marco Bevilacqua, Rohan Dharmakumar, Sotirios Tsaftaris BibTeX
DOI
Machine Learning Approaches for Enhanced, Shape Model Based 3D Image Segmentation
Unsupervised myocardial segmentation for cardiac MRI Medical Image Computing and Computer-Assisted Intervention -- MICCAI 2015, pp. 12-20, Vol.LNCS 9351, 2015 Anirban Mukhopadhyay, Ilkay Oksuz, Marco Bevilacqua, Rohan Dharmakumar, Sotirios Tsaftaris BibTeX
DOI
Machine Learning Approaches for Enhanced, Shape Model Based 3D Image Segmentation
2012
Automatic liver segmentation in contrast enhanced CT data using 3D free-form deformation based on optimal graph searching Master's thesis, Politecnico di Milano, Pietro Cerveri, Hans Lamecker (Advisors), 2012 Sara Orgiu BibTeX
Machine Learning Approaches for Enhanced, Shape Model Based 3D Image Segmentation
2011
Improvement of Image Segmentation Based on Statistical Shape and Intensity Models Master's thesis, Université libre de Bruxelles, Nadine Warzée, Olivier Debier, Hans Lamecker, Stefan Zachow (Advisors), 2011 Maximilien Renard BibTeX
Machine Learning Approaches for Enhanced, Shape Model Based 3D Image Segmentation