A multimodal deep learning-based dynamic prediction model for colorectal cancer liver metastasis

  • Haitao Zheng orcid

    Department of Ultrasound Medicine, First Affiliated Hospital of Hebei North University, Shijiazhuang 075000, China; First Clinical Medical College, Hebei North University, Shijiazhuang 075000, China
  • Dehui Wen orcid

    First Clinical Medical College, Hebei North University, Shijiazhuang 075000, China
  • Liwei Zhang orcid

    Department of Ultrasound Medicine, First Affiliated Hospital of Hebei North University, Shijiazhuang 075000, China
  • Haiyong Lu orcid

    Department of Ultrasound Medicine, First Affiliated Hospital of Hebei North University, Shijiazhuang 075000, China
  • Xiaoyu Li orcid

    Department of Ultrasound Medicine, First Affiliated Hospital of Hebei North University, Shijiazhuang 075000, China
  • Yongxin Li orcid

    Department of Thoracic Surgery, China Aerospace Science and Industry Corporation 731 Hospital, Beijing 100074, China
Article ID: 3902
DOI: https://doi.org/10.59400/adecp3902
Keywords: colorectal carcinoma, hepatic metastasis, contrast-enhanced ultrasound, multimodal fusion learning, discrete-time survival analysis, dynamic risk stratification

Abstract

Colorectal cancer liver metastasis (CRLM) remains a major determinant of long-term outcomes. Existing clinical models are typically static and single-modality, limiting early warning and individualized follow-up. We prospectively enrolled 300 treatment-naïve colorectal cancer patients. We collected preoperative three-phase contrast-enhanced ultrasound (CEUS) dynamic sequences, longitudinal serum marker measurements (EZH2/CD10) from preoperation through 12 months, and 35 clinical–pathological variables. The proposed Dynamic Modality Alignment Network (DMA-Net) includes (i) an imaging encoder based on an enhanced 3D-ResNet18 to extract perfusion kinetics, (ii) a molecular encoder using BiLSTM with temporal attention to model serial biomarkers, and (iii) a clinical encoder (MLP) for structured variables. A dynamic alignment module and cross-modal attention fuse modalities, followed by a discrete-time survival head that outputs month-specific conditional hazards and cumulative risks. On the held-out test set, the tri-modal model achieved an area under the curve (AUC) of 0.918 at 12 months with favorable calibration (Brier score 0.123), outperforming a traditional Cox model built from clinical variables (AUC 0.782, Brier score 0.177). Time-dependent evaluation showed stable AUCs from 3 to 12 months (0.904–0.919). Ablation experiments indicated that imaging and molecular branches contributed most to discrimination, whereas clinical variables improved calibration. Multimodal dynamic modeling integrating CEUS perfusion, longitudinal biomarkers, and clinical variables improves early warning and risk stratification for CRLM, and provides a practical framework to support personalized surveillance.

Published
2026-03-18
How to Cite
Zheng, H., Wen, D., Zhang, L., Lu, H., Li, X., & Li, Y. (2026). A multimodal deep learning-based dynamic prediction model for colorectal cancer liver metastasis. Advances in Differential Equations and Control Processes, 33(1). https://doi.org/10.59400/adecp3902
Issue
Vol. 33 No. 1 (2026)
Section
Article

Copyright (c) 2026 Haitao Zheng, Dehui Wen, Liwei Zhang, Haiyong Lu, Xiaoyu Li, Yongxin Li

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

References

[1]Sung H, Ferlay J, Siegel RL, et al. Global Cancer Statistics 2020: GLOBOCAN Estimates of Incidence and Mortality Worldwide for 36 Cancers in 185 Countries. CA: A Cancer Journal for Clinicians. 2021; 71(3): 209–249. doi: 10.3322/caac.21660

[2]Margonis GA, Sasaki K, Gholami S, et al. Genetic And Morphological Evaluation (GAME) score for patients with colorectal liver metastases. British Journal of Surgery. 2018; 105(9): 1210–1220. doi: 10.1002/bjs.10838

[3]Chen RJ, Lu MY, Wang J, et al. Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis. IEEE Transactions on Medical Imaging. 2022; 41(4): 757–770. doi: 10.1109/TMI.2020.3021387

[4]Cremolini C, Antoniotti C, Rossini D, et al. Upfront FOLFOXIRI plus bevacizumab and reintroduction after progression versus mFOLFOX6 plus bevacizumab followed by FOLFIRI plus bevacizumab in the treatment of patients with metastatic colorectal cancer (TRIBE2): A multicentre, open-label, phase 3, randomised, controlled trial. The Lancet Oncology. 2020; 21(4): 497–507. doi: 10.1016/S1470-2045(19)30862-9

[5]Ayez N, van der Stok EP, Grünhagen DJ, et al. The use of neoadjuvant chemotherapy in patients with resectable colorectal liver metastases: Clinical risk score as possible discriminator. European Journal of Surgical Oncology. 2015; 41(7): 859–867. doi: 10.1016/j.ejso.2015.04.012

[6]Lundberg SM, Lee SI. A Unified Approach to Interpreting Model Predictions. arXiv preprint. 2017; arXiv:1705.07874. doi: 10.48550/arXiv.1705.07874

[7]Simpson AL, Adams LB, Allen PJ, et al. Texture analysis of preoperative CT images for prediction of postoperative hepatic insufficiency: A preliminary study. Journal of the American College of Surgeons. 2015; 220(3): 339–346. doi: 10.1016/j.jamcollsurg.2014.11.027

[8]Dohan A, Gallix B, Guiu B, et al. Early evaluation using a radiomic signature of unresectable hepatic metastases to predict outcome in patients with colorectal cancer treated with FOLFIRI and bevacizumab. Gut. 2020; 69(3): 531–539. doi: 10.1136/gutjnl-2018-316407

[9]Li Y, Gong J, Shen X, et al. Assessment of primary colorectal cancer CT radiomics to predict metachronous liver metastasis. Frontiers in Oncology. 2022; 12: 861892. doi: 10.3389/fonc.2022.861892

[10]Lu MY, Chen TY, Williamson DFK, et al. AI-based pathology predicts origins for cancers of unknown primary. Nature. 2021; 594(7861): 106–110. doi: 10.1038/s41586-021-03512-4

[11]Dai Y, Gao Y, Liu F, et al. TransMed: Transformers Advance Multi-modal Medical Image Classification. Diagnostics. 2021; 11(8): 1384. doi: 10.3390/diagnostics11081384

[12]Moor M, Banerjee O, Abad ZSH, et al. Foundation models for generalist medical artificial intelligence. Nature. 2023; 616(7956): 259–265. doi: 10.1038/s41586-023-05881-4

[13]Park IJ, Choi GS, Lim KH, et al. Serum CEA monitoring after curative resection for colorectal cancer: Clinical significance of the preoperative level. Annals of Surgical Oncology. 2009; 16(11): 3087–3093. doi: 10.1245/s10434-009-0625-z

[14]Vale-Silva LA, Rohr K. Long-term cancer survival prediction using multimodal deep learning. Scientific Reports. 2021; 11: 13505. doi: 10.1038/s41598-021-92799-4

[15]Courtiol P, Maussion C, Moarii M, et al. Deep learning-based classification of mesothelioma improves prediction of patient outcome. Nature Medicine. 2019; 25(10): 1519–1525. doi: 10.1038/s41591-019-0583-3

[16]Chen J, Cheung HMC, Karanicolas PJ, et al. A radiomic biomarker for prognosis of resected colorectal liver metastases after preoperative chemotherapy using delayed-phase contrast-enhanced MRI. Frontiers in Oncology. 2023; 13: 898854. doi: 10.3389/fonc.2023.898854

[17]Luo H, Zhao Q, Wei W, et al. Circulating tumor DNA methylation profiles enable early diagnosis, prognosis prediction, and screening for colorectal cancer. Science Translational Medicine. 2020; 12(524): eaax7533. doi: 10.1126/scitranslmed.aax7533

[18]Han B, Zheng R, Zeng H, et al. Cancer incidence and mortality in China, 2022. Journal of the National Cancer Center. 2024; 4(1): 47–53. doi: 10.1016/j.jncc.2024.01.006

[19]Cheerla A, Gevaert O. Deep learning with multimodal representation for pancancer prognosis prediction. Bioinformatics. 2019; 35(14): i446–i454. doi: 10.1093/bioinformatics/btz342

[20]Van den Eynden GG, Majeed AW, Illemann M, et al. The multifaceted role of the microenvironment in liver metastasis: Biology and clinical implications. Cancer Research. 2013; 73(7): 2031–2043. doi: 10.1158/0008-5472.CAN-12-3931

[21]Holzinger A, Langs G, Denk H, et al. Causability and explainability of artificial intelligence in medicine. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery. 2019; 9(4): e1312. doi: 10.1002/widm.1312

[22]Jin C, Yu H, Ke J, et al. Predicting treatment response from longitudinal images using multi-task deep learning. Nature Communications. 2021; 12: 1851. doi: 10.1038/s41467-021-22188-y

[23]Fong Y, Fortner J, Sun RL, et al. Clinical Score for Predicting Recurrence after Hepatic Resection for Metastatic Colorectal Cancer: Analysis of 1001 Consecutive Cases. Annals of Surgery. 1999; 230(3): 309. doi: 10.1097/00000658-199909000-00004

[24]Gerds TA, Schumacher M. Consistent estimation of the expected Brier score in general survival models with right-censored data. Biometrical Journal. 2006; 48(6): 1029–1040. doi: 10.1002/bimj.200610301

[25]Cuturi M, Blondel M. Soft-DTW: A differentiable loss function for time-series. In: Proceedings of the 34th International Conference on Machine Learning. PMLR; 2017. pp. 894–903.

[26]Katzman JL, Shaham U, Cloninger A, et al. DeepSurv: personalized treatment recommender system using a Cox proportional hazards deep neural network. BMC Medical Research Methodology. 2018; 18: 24. doi: 10.1186/s12874-018-0482-1

[27]Ishwaran H, Kogalur UB, Blackstone EH, et al. Random survival forests. The Annals of Applied Statistics. 2008; 2(3): 841–860. doi: 10.1214/08-AOAS169

[28]Lee C, Zame WR, Yoon J, et al. DeepHit: a deep learning approach to survival analysis with competing risks. Proceedings of the AAAI Conference on Artificial Intelligence. 2018; 32(1). doi: 10.1609/aaai.v32i1.11842

[29]Lee C, Yoon J, van der Schaar M. Dynamic-DeepHit: A deep learning approach for dynamic survival analysis with competing risks based on longitudinal data. IEEE Transactions on Biomedical Engineering. 2020; 67(1): 122–133. doi: 10.1109/TBME.2019.2909027

[30]Tanner J, Sharples LD, Daniel RM, et al. Dynamic survival prediction combining landmarking with a machine learning ensemble: methodology and empirical comparison. Journal of the Royal Statistical Society: Series A (Statistics in Society). 2021; 184(1): 3–30. doi: 10.1111/rssa.12611

[31]Mesinovic M, Watkinson P, Zhu T. DySurv: A deep survival model for dynamic prediction based on longitudinal data. Journal of the American Medical Informatics Association. 2024; 33(1): 112–122. doi: 10.1093/jamia/ocae271

[32]Nikolaou N, Salazar D, RaviPrakash H, et al. A machine learning approach for multimodal data fusion for survival prediction in cancer patients. npj Precision Oncology. 2025; 9: 128. doi: 10.1038/s41698-025-00917-6

[33]Li Y, Daho MEH, Conze PH, et al. A review of deep learning-based information fusion techniques for multimodal medical image classification. Computers in Biology and Medicine. 2024; 177: 108635. doi: 10.1016/j.compbiomed.2024.108635

[34]Brooks JA, Kallenbach M, Radu IP, et al. Artificial Intelligence for Contrast-Enhanced Ultrasound of the Liver: A Systematic Review. Digestion. 2025; 106(3): 227–244. doi: 10.1159/000541540

[35]Shabani N, Yaseri M, Alimi R, et al. Dynamic survival analysis via a landmarking-gradient boosting approach and its application to kidney transplant data. BMC Medical Informatics and Decision Making. 2025; 25(1): 368. doi: 10.1186/s12911-025-03205-2

[36]Rizopoulos D, Taylor JMG. Optimal estimation of dynamic predictions from joint models using super learning. Statistics in Medicine. 2024; 43(7): 1315–1328. doi: 10.1002/sim.10010