Applying Transformer-Based Deep Learning Model for Predicting Multimorbidity in Older Adults
Date
2023-09-15
Authors
Journal Title
Journal ISSN
Volume Title
Publisher
ORCID
0000-0001-8537-2601
Type
Thesis
Degree Level
Masters
Abstract
Disease predictive modelling supports decision-making for policymakers and healthcare providers, and aids in managing health conditions for individuals. Recent advances in deep learning have exhibited outstanding accuracy in diagnosis prediction. Bidirectional Encoder Representations from Transformers (BERT) has been applied to clinical research in disease prediction tasks due to its ability to comprehend entire diagnosis histories as sequences. However, there have been few studies conducted on the use of BERT models utilizing structured medical datasets for disease prediction. In particular, the increasing prevalence of multimorbidity is becoming a burden for the geriatric population, leading to adverse outcomes such as mortality, disability, and frequent healthcare utilization. Limited research exists that constructs Transformer models to predict multimorbidity, or that combines covariates in a multi-modal approach. This thesis aims to address these research gaps by constructing a Transformer pre-training model and fine-tuning model to predict multi-label outcomes of multimorbidity based on sequence-based diagnosis data and risk factors. Consequently, the pre-trained and fine-tuned models that can predict multimorbidity in the geriatric population were developed.
In this thesis, longitudinal data from the Korean Health Panel Survey (KHPS) was used for analysis. The data, collected by the Korea Institute for Health and Social Affairs and the National Health Insurance Service from 2008 to 2018, comprises up to 11 interviews. Older adults aged 60 or above (N=7,667) were selected for predicting multimorbidity, which was defined as having two or more chronic conditions. Age-related 60 chronic conditions were classified, and the twelve most prevalent chronic diseases were selected for prediction: hypertension, musculoskeletal and joint diseases, inflammatory arthropathies, diseases of the esophagus/stomach/duodenum, dorsopathies, diabetes, ear/nose/throat diseases, dyslipidemia, osteoporosis, eye diseases, peripheral neuropathy, and colitis/related diseases. Explanatory variables-such as sex, age, Body Mass Index (BMI), life insurance, and income quintile-were selected based on their significance in the multivariable Generalized Estimating Equations (GEE) model. Pre-training was conducted by building a Masked Language Model (MLM) to capture disease representations. The Medical Information Mart for Intensive Care (MIMIC-III) dataset was employed as an external validation dataset to test the pre-trained model. Metrics such as precision, recall, F1-score, average precision, and the Area Under the Receiver Operating Characteristic curve (AUROC) were used to present experimental results. The fine-tuned model and the scratch-trained model were compared using both the KHPS and MIMIC-III datasets. In the fine-tuning stage, multi-label classification was conducted to predict multiple chronic conditions 1 month and 1 year after a certain follow-up point in older adults. The performance of the fine-tuned model was compared to machine learning models such as binary relevance, classifier chain, and label powerset. Lastly, our final model was compared to existing Transformer multi-label classification models, BEHRT and Med-BERT.
The use of a pre-trained model improved macro precision by 7.9% and macro recall by 1.3% compared to the scratch-trained model when the validation dataset, MIMIC-III, was utilized. Incorporating variables combination exhibited no significant differences. Macro precision values for Exp1 (sex, age, BMI, and income), Exp2 (sex, age, and BMI), and Exp3 (sex and age) were 0.431, 0.428, 0.436, and 0.431, respectively, indicating minimal differences. The binary relevance achieved approximately 9.3% higher macro precision but exhibited a 24.8% lower macro recall compared to our model. When compared to the BEHRT model, our model had 12.6% lower macro precision and 15.1% higher macro recall.
These findings indicate that a pre-training model has the potential to improve a model's performance when applied to datasets with similar characteristics. Overall, our model exhibited a higher recall than the multi-label machine learning methods and the deep Transformer models, BEHRT and Med-BERT. Furthermore, this thesis provided a model capable of predicting multimorbidity 1 month and 1 year after a certain follow-up point. With robust validation and improved precision, recall, and F1-score, our model can assist both individuals and medical experts in managing health conditions and medical costs in the future.
Description
Keywords
Transformer, Deep Learning, Artificial Intelligence, Multimorbidity
Citation
Degree
Master of Science (M.Sc.)
Department
School of Public Health
Program
Biostatistics