Temporal Explanation Drift: Measuring and analyzing temporal explanation drift in deep neural networks
-
Zaryab Rahman
Department of Computer Science and IT, Faculty of Computing Sciences and Engineering, University of Malakand, Chakdara 18800, Pakistan
-
Mattia Ottoborgo
Trustpilot, 1112 Copenhagen, Denmark
Abstract
The evaluation of explainable artificial intelligence (XAI) methods has largely focused on the properties of a single, fully-trained model. This static view overlooks a critical dimension of reliability: the stability of an explanation throughout the training process. In this paper, we introduce a formal framework to measure and analyze Temporal Explanation Drift (TED), quantifying how a model’s explanations for a consistent, correctly classified input change from one training epoch to the next. Our analysis on Vision Transformers reveals a fundamental decoupling of stability: the high-level semantic focus of an explanation stabilizes early, while its fine-grained structural representation continues to drift long after predictive accuracy has converged. We demonstrate that this is a general phenomenon that extends to Convolutional Neural Networks (CNNs), but find that its magnitude is critically dependent on the explanation method. Our experiments reveal that post-hoc, gradient-based methods like Grad-CAM are dramatically less stable than intrinsic methods like attention rollout. A final, controlled experiment scientifically dissects these effects, showing that the choice of explanation method is the primary driver of temporal instability, which is further modulated by a nuanced interaction with the model’s architecture. Our work establishes temporal stability as a distinct and vital property for evaluating the trustworthiness of explanations and provides a practical framework for its measurement.
Copyright (c) 2025 Zaryab Rahman, Mattia Ottoborgo
This work is licensed under a Creative Commons Attribution 4.0 International License.
References
[1]Litjens G, Kooi T, Bejnordi BE, et al. A survey on deep learning in medical image analysis. Medical Image Analysis. 2017; 42: 60–88.
[2]Esteva A, Kuprel B, Novoa RA, et al. Dermatologist-level classification of skin cancer with deep neural networks. Nature. 2017; 542(7639): 115–118.
[3]Bejnordi BE, Veta M, Van Diest PJ, et al. Diagnostic assessment of deep learning algorithms for detection of lymph node metastases in women with breast cancer. JAMA. 2017; 318(22): 2199–2210.
[4]Bojarski M, Del Testa D, Dworakowski D, et al. End to end learning for self-driving cars. arXiv preprint. 2016. doi: 10.48550/arXiv.1604.07316
[5]Grigorescu S, Trasnea B, Cocias T, et al. A survey of deep learning techniques for autonomous driving. Journal of Field Robotics. 2020; 37(3): 362–386.
[6]Levine S, Finn C, Darrell T, et al. End-to-end training of deep visuomotor policies. Journal of Machine Learning Research. 2016; 17(1): 1334–1373.
[7]Rudin C. Stop explaining black box machine learning models for high stakes decisions and use interpretable models instead. Nature Machine Intelligence. 2019; 1(5): 206–215.
[8]Goodfellow IJ, Shlens J, Szegedy C. Explaining and Harnessing Adversarial Examples. arXiv preprint. 2014. doi: 10.48550/arXiv.1412.6572
[9]Gunning D, Stefik M, Choi J, et al. XAI—Explainable artificial intelligence. Science Robotics. 2019; 4(37).
[10]Arrieta AB, Díaz-Rodríguez N, Del Ser J, et al. Explainable Artificial Intelligence (XAI): Concepts, taxonomies, opportunities and challenges toward responsible AI. Information Fusion. 2020; 58: 82–115.
[11]Abnar S, Zuidema W. Quantifying attention flow in transformers. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; 5-10 July 2020; Online. pp. 4190–4197.
[12]Chefer H, Gur S, Wolf L. Transformer Interpretability Beyond Attention Visualization. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR); 20-25 June 2021; Nashville, TN, USA. pp. 782–791.
[13]Selvaraju RR, Cogswell M, Das A, et al. Grad-CAM: Visual explanations from deep networks via gradient-based localization. In: Proceedings of the International Conference on Computer Vision (ICCV); 22-29 October 2017; Venice, Italy. pp. 618–626.
[14]Jacovi A, Goldberg Y. Towards faithfully interpretable NLP systems: How should we define and evaluate faithfulness? In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics; 5-10 July 2020; Online. pp. 4198–4205.
[15]Ghorbani A, Abid A, Zou J. Interpretation of neural networks is fragile. Proceedings of the AAAI Conference on Artificial Intelligence. 2019; 33(1): 3681–3688.
[16]Dombrowski AK, Alber M, Anders CJ, et al. Explanations can be manipulated and geometry is to blame. In: Proceedings of the 33rd International Conference on Neural Information Processing Systems (NeurIPS 2019); 8-14 December 2019; Vancouver, BC, Canada.
[17]Kindermans PJ, Hooker S, Adebayo J, et al. The (un)reliability of saliency methods. In: Explainable AI: Interpreting, Explaining and Visualizing Deep Learning. Springer; 2019. pp. 267–280.
[18]Adebayo J, Gilmer J, Muelly M, et al. Sanity checks for saliency maps. In: Proceedings of the 32nd Conference on Neural Information Processing Systems (NeurIPS 2018); 2-8 December 2018; Montréal, QC, Canada. pp. 9505–9515.
[19]Nie W, Zhang Y, Patel AB. A theoretical explanation for perplexing behaviors of backpropagation-based visualizations. In: Proceedings of the 35th International Conference on Machine Learning (ICML); 10–15 July 2018; Stockholm, Sweden. pp. 3809–3818.
[20]Dosovitskiy A, Beyer L, Kolesnikov A, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint. 2020. doi: 10.48550/arXiv.2010.11929
[21]Touvron H, Cord M, Douze M, et al. Training data-efficient image transformers & distillation through attention. Proceedings of the 38th International Conference on Machine Learning. 2021; 139: 10347–10357.
[22]Khan S, Naseer M, Hayat M, et al. Transformers in Vision: A Survey. ACM Computing Surveys (CSUR). 2022; 54(10s): 1–41. doi: 10.1145/3505244
[23]Koh PW, Nguyen T, Tang YS, et al. Concept Bottleneck Models. Proceedings of the 37th International Conference on Machine Learning. 2020; 119: 5338–5348.
[24]Howard A, Sandler M, Chu G, et al. Searching for mobilenetv3. In: Proceedings of the International Conference on Computer Vision (ICCV); 27 October-2 November 2019; Seoul, Republic of Korea. pp. 1314–1324.
[25]Ribeiro MT, Singh S, Guestrin C. “Why Should I Trust You?”: Explaining the Predictions of Any Classifier. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining; 13-17 August 2016; San Francisco, CA, USA. pp. 1135–1144.
[26]Lundberg SM, Lee SI. A Unified Approach to Interpreting Model Predictions. In: Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS 2017); 4-9 December 2017; Long Beach, CA, USA. pp. 4768-4777.
[27]Sundararajan M, Taly A, Yan Q. Axiomatic Attribution for Deep Networks. Proceedings of the 34th International Conference on Machine Learning. 2017; 70: 3319-3328.
[28]Samek W, Montavon G, Lapuschkin S, et al. Explaining Deep Neural Networks and Beyond: A Review of Methods and Applications. Proceedings of the IEEE. 2021; 109(3): 247–278.
[29]Bodria F, Giannotti F, Guidotti R, et al. Benchmarking and Survey of Explanation Methods for Black Box Models. Data Mining and Knowledge Discovery. 2023; 37(5): 1719–1778.
[30]Hooker S, Erhan D, Kindermans PJ, et al. A Benchmark for Interpretability Methods in Deep Neural Networks. In: Proceedings of the 33rd Conference on Neural Information Processing Systems (NeurIPS 2019); 8-14 December 2019; Vancouver, BC, Canada.
[31]Hedström A, Weber L, Bareeva D, et al. Quantus: An Explainable AI Toolkit for Responsible Evaluation of Neural Network Explanations and Beyond. Journal of Machine Learning Research. 2023; 24: 1–11.
[32]Agarwal C, Krishna S, Saxena E, et al. OpenXAI: Towards a transparent evaluation of model explanations. In: Proceedings of the 36th International Conference on Neural Information Processing Systems (NeurIPS 2022); 28 November-9 December 2022; New Orleans, LA, USA.
[33]Raghu M, Gilmer J, Yosinski J, et al. Svcca: Singular vector canonical correlation analysis for deep learning dynamics and interpretability. In: Proceedings of the 31st International Conference on Neural Information Processing Systems (NIPS 2017); 4-9 December 2017; Long Beach, CA, USA. pp. 6076–6085.
[34]Kornblith S, Norouzi M, Lee H, et al. Similarity of neural network representations revisited. Proceedings of the 36th International Conference on Machine Learning. 2019; 97: 3519–3529.
[35]Nanda N, Chan L, Lieberum T, et al. Progress Measures for Grokking via Mechanistic Interpretability. arXiv preprint. 2023. doi: 10.48550/arXiv.2301.05217
[36]Gururangan S, Marasović A, Swayamdipta S, et al. Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models. arXiv preprint. 2022. doi: 10.48550/arXiv.2208.03306
[37]Frankle J, Dziugaite GK, Roy DM, et al. Linear Mode Connectivity and the Lottery Ticket Hypothesis. Proceedings of the 37th International Conference on Machine Learning. 2020; 119: 3259–3269.
[38]Zhang C, Bengio S, Hardt M, et al. Understanding Deep Learning (Still) Requires Rethinking Generalization. Communications of the ACM. 2021; 64(3): 107–115.
[39]Lin J. Divergence Measures Based on the Shannon Entropy. IEEE Transactions on Information Theory. 1991; 37(1): 145–151.
[40]Wang Z, Bovik AC, Sheikh HR, et al. Image Quality Assessment: From Error Visibility to Structural Similarity. IEEE Transactions on Image Processing. 2004; 13(4): 600–612.
