Tissue Classification Based on Machine Learning
Machine learning methods based on deep neural networks (DNNs) have already proven their strength in the robust semantic segmentation of image data in medicine and are successfully used, for example, for the classification of malignant tissue in mammograms. Compared to classical pattern recognition methods, no explicit features are specified, instead they are learned based on a large data sets with known segmentation/labelling and stored as weights of the DNN.
In this project, one challenge is to merge different types of established pre- and intraoperative data, like MRI scans and endoscopic images, with data that is only sparsley available at certain points, like the signals generated by the sensor from the A projects or the labels defined by the intraoperative histopathological frozen section.
With the help of transfer learning between the modalities, the intraoperatively acquired camera image should also be processed in real time in such a way that a spatial registration with the other multimodal models can take place.
Linkage of Multimodal Information with Databases
In tumor diagnostics there are some publicly available image data sets (BACH 2018, CAMELYON16 2016, CAMELYON17 2018), which are already classified and can be used for training DNNs. However, in most cases only one modality is given, e.g. histological sections stained with hematoxylin-eosin (HE). In this subproject, DNN-based procedures will be investigated that support both the transfer of the classification and the collaborative training on multiple modalities.
The uneven distribution of labeled data poses a further challenge. Thus, segmentation maps, i.e. classifications for each pixel for histological sections or radiological images, exist only in some cases. Image data with findings, on the other hand, are available more frequently. These labels are much easier to produce and also easier to collect for the newly developed modalities from subprojects A1-A5.
The aim is therefore not only to bring the modalities together, but also to unify the heterogeneity of the available label data by using semi- or non-supervised learning procedures.