Page 92 - AIH-1-4

Artificial Intelligence in Health Transformer-based radiology report summaries

reports. Therefore, summarizing these reports not only trained BERT-based models. They developed
aids in the clinical workflow but also enhances patients’ ClinicalBioBERTSum, which incorporates domain-
understanding of their medical conditions and improves specific BERT models into the BERTSum architecture. This
their engagement with health-care personnels. model was applied to the MIMIC-CXR dataset, focusing
In addition, although there is an evident need for on predicting the “Impression” section of radiology reports
effective summarization tools in radiology, advancements based on the “Indication” and “Findings” sections. Their
in NLP have been slow to penetrate this domain. The model achieved a recall-oriented understudy for gisting
application of state-of-the-art NLP models to radiology evaluation (ROUGE)-L F1 score of 57.37 and introduced
report summarization remains relatively unexplored. ClinicalBioBERTScore to better evaluate the semantic
Addressing this gap presents a significant opportunity for quality of the summaries. Their work emphasizes the
innovation and improvement in medical communication importance of domain-specific pre-training and fine-
and patient care. tuning in improving summarization performance for
clinical texts. By leveraging ClinicalBioBERT, a model
The present study addresses this gap by developing fine-tuned on MIMIC-III clinical texts, they improved the
and evaluating a novel summarization model tailored representation of medical semantics in radiology reports.
specifically for radiology reporting. By utilizing the MIMIC The study also explored the use of custom tokenizers
– chest X-ray (CXR) dataset, we fine-tuned a bidirectional and a two-stage fine-tuning process, which showed that
3
encoder representation from transformers BERT-based combining extractive and abstractive summarization
model to create Biomedical-BERT2BERT, achieving objectives could enhance model performance.
2
state-of-the-art performance in generating concise and 2
accurate summaries of radiology findings. Our approach Devlin et al. introduced BERT, a novel transformer-
not only leverages advanced NLP techniques but also based model pre-trained on large text corpora for various
introduces a novel data augmentation strategy to enhance NLP tasks. BERT’s bidirectional training allows it to capture
model performance. This model takes multiple free-text context from both left and right surroundings, significantly
radiology report fields as input and uses a sequence-to- enhancing its performance on tasks such as question
sequence architecture to output abstract summaries. The answering, language inference, and more. The model sets
main contributions of our work are as follows: new benchmarks on several NLP tasks, demonstrating the
(1) We developed Biomedical-BERT2BERT, an adjusted effectiveness of pre-training on large datasets followed by
BERT-based model with state-of-the-art performance fine-tuning on specific tasks. The study’s impact on the field
in radiology text summarization, assisting radiologists is profound, as BERT has become a foundational model
in generating concise impressions from reports for many subsequent NLP advancements. Its architecture,
(2) We introduced a novel data augmentation strategy to which utilizes multiple layers of transformers and a
improve performance on related tasks with MIMIC- masked language model objective, enables it to learn deep
CXR reports contextual representations. This work paved the way for
(3) We conducted an in-depth analysis of Biomedical- numerous domain-specific adaptations, such as BioBERT
BERT2BERT’s performance, knowledge gain, and and ClinicalBERT, which tailor the model to specific fields
limitations with respect to disease distribution and by further pre-training on domain-specific texts.
other architectures. Alsentzer et al. explored the application of contextual
6
Our data processing pipeline and model training code word embedding models, such as ELMo and BERT, to
are also provided. 4 clinical text, addressing the gap in publicly available pre-
trained models for clinical NLP tasks. They developed and
2. Related works released BERT models specifically trained on clinical text,
demonstrating significant improvements over traditional
Initial work in this domain was conducted by Chen et al., BERT and BioBERT models on several clinical NLP tasks.
5
who reported promising results in predicting radiologist These tasks included named entity recognition (NER) and
impressions from raw findings using fine-tuned BERT- medical natural language inference, where their domain-
based encoder–decoder models. We extended this work by specific models outperformed general domain models. The
experimenting with different architectures, understanding study highlights the challenges of applying general pre-
the limitations of applying language models in this trained models to domain-specific texts due to linguistic
domain, and investigating the effectiveness of modern differences. By training BERT models on the MIMIC-
linear attention mechanisms on MIMIC-CXR. III dataset, which comprises approximately two million
Chen et al. addressed the challenge of abstractive clinical notes, they tailored the embeddings to better fit the
5
summarization in radiology reporting using pre- clinical context. This work is notable for providing publicly

Volume 1 Issue 4 (2024) 86 doi: 10.36922/aih.3846