Page 55 - ARNM-1-2

Advances in Radiotherapy
& Nuclear Medicine Image fusion’s significance in medicine

structure in the cartoon components and a sparse- fusion needs. For example, one can utilize classic CNN
representation-based method for the texture components. structures such as VGG and ResNet, or explore custom
Their approach outperformed the state-of-art methods in network structures and convolutional kernel sizes to achieve
[26]
both visual and quantitative evaluations. Daneshvar and more precise feature extraction . In addition, multi-scale
Ghassemian combined the IHS method with the retina- feature fusion can be used to fuse feature maps of different
[22]
inspired model (RIM) fusion technique, which enabled scales, thereby enhancing the accuracy and robustness of
the preservation of more spatial features and functional the fusion image. Alternatively, pixel-level fusion methods
information content, respectively. By integrating the can be used to fuse information for each pixel to generate
advantages of both IHS and RIM fusion methods, their a more refined fusion image. Overall, CNN-based image
algorithm successfully improved both functional and fusion presents a highly promising image fusion method
spatial information content. for achieving more accurate and efficient medical image
fusion. Continuous exploration of new network structures,
3. Deep learning-based image fusion hyperparameters, and fusion methods is essential for
methods further improving its performance .
[27]
The emergence of deep learning has further advanced the Lahoud and Süsstrunk proposed a real-time image
[28]
development of image fusion technology. Deep learning- fusion method that utilizes pre-trained neural networks
based image fusion method utilizes deep neural network to generate a single image containing features from multi-
models to learn image features and weights, achieving modal sources. In contrast, Teng et al. presented a fusion
[29]
automatic image fusion. First, multiple modal medical algorithm based on neuro-fuzzy logic and utilized a hybrid
images are input into the deep neural network, which algorithm that combines the back propagation algorithm
captures key information by extracting and learning with the least mean square (LMS) algorithm to train the
features through multiple layers. Subsequently, the parameters of the membership function. The fused images
extracted features are fused using a fusion layer to generate based on neuro-fuzzy logic not only reserve more texture
the fused image. Finally, through training and optimizing features but also enhance the information characteristics
the network weights, the fused image achieves the highest of the two original images. Wang et al. proposed an
[30]
quality and information preservation. algorithm that utilizes a trained Siamese CNN to fuse the
Deep learning offers new feature representation pixel activity information of source images, enabling the
methods, addressing the limitations of multiscale and generation of a weight map. In addition, they implemented
spatial variability present in traditional methods. The a contrast pyramid to decompose the source image.
main deep learning-based fusion methods include 3.2. CSR-based image fusion
convolutional neural networks (CNN), convolutional
sparse representation (CSR), generative adversarial The core idea of CSR-based image fusion technology is to
networks (GAN), and deep restricted Boltzmann machines leverage the sparsity of CNN to decompose image features
(RBM) . In Section 3, we focus on the application of into a set of sparse coefficients and then fuse them by
[23]
deep learning-based image fusion methods in the field of weighted summation. First, CNN is used to extract the
NMMI. Table 2 provides an overview of the works that features of two images to be fused, and subsequently, the
report on the deep learning-based image fusion methods extracted features are decomposed to obtain the sparse
in NMMI. coefficients for each feature. For each pixel, the sparse
coefficients of the two images are weighted and summed
3.1. CNN-based image fusion based on their feature weights in the two images, resulting
CNN-based image fusion is a common method for in new sparse coefficients. Finally, these new sparse
medical image fusion. CNNs learn features from raw data, coefficients are restored into an image to obtain the fused
and thus learn different types of image features, ultimately image. CSR-based image fusion technology effectively
achieving image fusion . In this approach, different types retains the main features of both images during the fusion
[24]
of medical images are first extracted for features through process while demonstrating robust detail preservation
[31]
CNNs, resulting in feature representations of different ability and noise resistance .
types of images. Then, these feature representations are Liu et al. designed a novel diagnostic framework
[32]
fused to generate the final fusion image . The flexibility with deep learning architecture to aid the diagnosis of
[25]
of CNNs allows for the use of different network structures Alzheimer’s disease (AD). This framework employs a zero-
and hyperparameters during the feature extraction process, masking strategy for data fusion, extracting complementary
enabling adaptation to various types of medical images and information from multiple data modalities. Compared

Volume 1 Issue 2 (2023) 5 https://doi.org/10.36922/arnm.0870