Page 121 - AIH-1-2

Artificial Intelligence in Health Medical instruction-tuning for Japanese LLMs

on Academic Articles in Medical Sciences (in Japanese). In: doi: 10.1038/sdata.2016.35
Proceedings of the 29 Annual Meeting of the Association for 18. Kawazoe Y, Shibata D, Shinohara E, Aramaki E, Ohe K.
th
Natural Language Processing; 2023. A clinical specific BERT developed using a huge Japanese
6. Devlin J, Chang MW, Lee K, Toutanova K. BERT: Pre- clinical text corpus. PLoS One. 2021;16(11):e0259763.
training of Deep Bidirectional Transformers for Language doi: 10.1371/journal.pone.0259763
Understanding. In: Proceedings of the 2019 Conference of the
North American Chapter of the Association for Computational 19. Vaswani A, Shazeer N, Parmar N, et al. Attention is all you
Linguistics: Human Language Technologies, Volume 1 (Long need. Adv Neural Inf Process Syst. 2017;30:5998-6008.
and Short Papers); 2019. 20. Bolton E, Hall D, Yasunaga M, Lee T, Manning C, Liang P.
7. Hu EJ, Wallis P, Allen-Zhu Z, et al. LoRA: Low-rank Stanford CRFM Introduces PubMedGPT 2.7B; 2022.
Adaptation of Large Language Models. In: International Available from: https://hai.stanford.edu/news/stanford-
Conference on Learning Representations; 2021. crfm-introduces-pubmedgpt-27b [Last accessed on 2024
Apr 04].
8. Dettmers T, Pagnoni A, Holtzman A, Zettlemoyer L.
QLoRA: Efficient Finetuning of Quantized LLMs. Advances 21. Luo R, Sun L, Xia Y, et al. BioGPT: Generative pre-trained
in Neural Information Processing Systems. 2023;36:10088- transformer for biomedical text generation and mining.
10115. Brief Bioinform. 2022;23:bbac409.
9. Suzuki M, Hirano M, Sakaji H. From Base to Conversational: doi: 10.1093/bib/bbac409
Japanese Instruction Dataset and Tuning Large Language 22. Luo Y, Zhang J, Fan S, et al. BioMedGPT: Open Multimodal
Models. In: 2023 IEEE International Conference on Big Data Generative Pre-trained Transformer for Biomedicine.
(Big Data); 2023. arXiv:2308.09442 [arXiv Preprint], 2023.
10. Xie Q, Han W, Zhang X, et al. PIXIU: A Comprehensive doi: 10.48550/arXiv.2308.09442
Benchmark, Instruction Dataset and Large Language Model 23. Radford A, Wu J, Child R, Luan D, Amodei D, Sutskever I.
for Finance. Advances in Neural Information Processing Language models are unsupervised multitask learners.
Systems. 2023;36:33469-33484. OpenAI Blog. 2019;1:9.
11. Zhou C, Liu P, Xu P, et al. Lima: Less is More for Alignment. 24. Touvron H, Martin L, Stone K, et al. Llama 2: Open
Advances in Neural Information Processing Systems. Foundation and Fine-tuned Chat Models. arXiv:2307.09288
2023;36:55006-55021. [arXiv Preprint], 2023.
12. Brown T, Mann B, Ryder N, et al. Language models are doi: 10.48550/arXiv.2307.09288
few-shot learners. Adv Neural Inf Process Syst. 2020;
33:1877-1901. 25. Wei J, Bosma M, Zhao V, et al. Fine-tuned Language Models
are Zero-shot Learners. In: International Conference on
13. Lee J, Yoon W, Kim S, et al. BioBERT: A pre-trained Learning Representations; 2022.
biomedical language representation model for biomedical
text mining. Bioinformatics. 2020;36(4):1234-1240. 26. Mangrulkar S, Gugger S, Debut L, Belkada Y, Paul S. PEFT:
State-of-the-art Parameter-Efficient Fine-tuning Methods;
doi: 10.1093/bioinformatics/btz682 2022. Available from: https://github.com/huggingface/peft
14. Rasmy L, Xiang Y, Xie Z, Tao C, Zhi D. Med-BERT: [Last accessed on 2024 Apr 04].
Pretrained contextualized embeddings on large-scale 27. Dettmers T, Zettlemoyer L. The Case for 4-bit Precision:
structured electronic health records for disease prediction. K-bit Inference Scaling Laws. In: International Conference
NPJ Digit Med. 2021;4:86. on Machine Learning. PMLR; 2023.
doi: 10.1038/s41746-021-00455-y 28. Jin D, Pan E, Oufattole N, Weng WH, Fang H, Szolovits P.
15. Huang K, Altosaar J, Ranganath R. ClinicalBERT: Modeling What disease does this patient have? A large-scale open
Clinical Notes and Predicting Hospital Readmission. domain question answering dataset from medical exams.
arXiv:1904.05342 [arXiv Preprint], 2019. Appl Sci. 2021;11(14):6421.
doi: 10.3390/app11146421
doi: 10.48550/arXiv.1904.05342
29. Pal A, Umapathi LK, Sankarasubbu, M. MedMCQA:
16. Gu Y, Tinn R, Cheng H, et al. Domain-specific language A Large-scale Multi-Subject Multi-Choice Dataset for
model pretraining for biomedical natural language Medical Domain Question Answering. In: Proceedings of the
processing. ACM Trans Comput Healthc. 2021;3(1):1-23.
Conference on Health, Inference, and Learning (2022); 2022.
doi: 10.1145/3458754 p. 248-260.
17. Johnson AEW, Pollard TJ, Shen L, et al. MIMIC-III, a freely 30. Jin Q, Dhingra B, Liu Z, Cohen WW, Lu X. PubMedQA: A
accessible critical care database. Sci Data. 2016;3:160035. Dataset for Biomedical Research Question Answering. In:

Volume 1 Issue 2 (2024) 115 doi: 10.36922/aih.2695