Model validation based on value-of-information theory

  • George Hazelrigg Department of Mechanical Engineering, George Mason University, Fairfax, VA 22030, USA
Article ID: 2992
DOI: https://doi.org/10.59400/jam2992
Keywords: decision theory; value of information; model validation; systems modeling

Abstract

The modeling and simulation community has devoted considerable attention to the question of model validity as a condition for the use of a model in an engineering decision-making process. Their work has focused on the concept of “accuracy”, loosely defined as the difference between a model-computed result and a real-world result. The objective of this paper is to introduce an alternative approach, based on classical decision theory, that focuses on the value of the information that a model provides to the decision-making process. This is a significant departure from the current approach to model validation, and it derives from the preference, “I want the best outcome that I can get”. Use is made of an example case that results in a paradox to illustrate weaknesses in the accuracy-focused approach. Instead of advocating the use of a model based on its accuracy, this work advocates using a model if it adds value to the overall application, thus relating validation directly to system performance. The approach fills significant gaps in the current theory, notably providing a clearly defined validity metric and a mathematically rigorous rationale for the use of this metric.

Published
2025-07-02
How to Cite
Hazelrigg, G. (2025). Model validation based on value-of-information theory. Journal of AppliedMath, 3(4), 2992. https://doi.org/10.59400/jam2992
Issue
Vol. 3 No. 4 (2025)
Section
Article

Copyright (c) 2025 Author(s)

Creative Commons License

This work is licensed under a Creative Commons Attribution 4.0 International License.

References

[1]Sargent RG, Balci O. History of verification and validation of simulation models. In: Proceedings of the 2017 Winter Simulation Conference; 3–6 December 2017; Las Vegas, NV, USA. pp. 292–307.

[2]National Research Council. Assessing the Reliability of Complex Models: Mathematical and Statistical Foundations of Verification, Validation, and Uncertainty Quantifi-cation. National Academies Press; 2012.

[3]Oberkampf WL, Deland SM, Rutherford B, et al. Estimation of total uncertainty in modeling and simulation. Sandia National Laboratories; 2000.

[4]Easterling RG. Measuring the predictive capability of computational models: Principles and methods, issues and illustrations. Sandia National Laboratories; 2001.

[5]Trucan TG, Easterling RG, Dowding KJ, et al. Description of the Sandia Validation Metrics Project. Sandia National Laboratories; 2001.

[6]Hills RG, Trucano TG. Statistical validation of engineering and scientific models: A maximum likelihood based metric. Sandia National Laboratories; 2002.

[7]Sentz K, Ferson S. Combination of evidence in Dempster-Shafer theory. Sandia National Laboratories; 2002.

[8]Ferson S, Kreinovich V, Ginzburg L, et al. Constructing probability boxes and Dempster-Shafer structures. Sandia National Laboratories; 2003.

[9]Hills RG, Leslie IH. Statistical validation of engineering and scientific models: Validation experiments to application. Sandia National Laboratories; 2003.

[10]Oberkampf WL, Trucano TG, Hirsch C. Verification, validation, and predictive capability in computational engineering and physics. Sandia National Laboratories; 2003.

[11]Ferson S, Hajagos J, Berleant D, et al. Dependence in probabilistic modeling, Dempster-Shafer theory, and probability bounds analysis. Sandia National Laboratories; 2004.

[12]Helton JC, Johnson JD, Oberkampf WL, Storlie CB. A sampling-based computational strategy for the representation of epistemic uncertainty in model predictions with evidence theory. Sandia National Laboratories; 2006.

[13]Oberkampf WL, Pilch M, Trucano TG. Predictive capability maturity model for computational modeling and simulation. Sandia National Laboratories; 2007.

[14]Fishman GS, Kiviat PJ. The statistics of discrete-event simulation. Simulation. 1968; 10(4): 185–195.

[15]MITRE. Systems engineering guide. MITRE; 2014.

[16]Toptan A, Porter NW, Hales JD, et al. Construction of a code certification matrix for heat conduction with finite element code applications. Journal of Verification Validation and Uncertainty Quantification. 2020; 5(4): 041002–041017.

[17]Howard RA. Information value theory. IEEE Transactions on Systems Science and Cybernetics, SSC. 1966; 2(1): 22–26.

[18]Hillier FS, Lieberman GJ. Introduction to Operations Research, 5th ed. McGraw-Hill; 1990.

[19]Howard RA, Abbas AE. Foundations of Decision Analysis. Pearson Publishing; 2016.

[20]Hazelrigg GA, Saari DG. Toward a theory of systems engineering. Journal of Mechanical Design. 2022; 144: 011402–011410.

[21]Hazelrigg GA. Fundamentals of Decision Making for Engineering Design and Systems Engineering. NEILS CORP; 2012.

[22]Hu KT, Orient GE. The 2014 Sandia verification and validation challenge: Problem statement. Journal of Verification, Validation and Uncertainty Quantification. 2016; 1: 011001–011011.

[23]Beghini LL, Hough PD. Sandia verification and validation challenge problem: A PCCM based approach to assessing prediction credibility. Journal of Verification, Validation and Uncertainty Quantification. 2016; 1: 011002–011012.

[24]Choudhary A, Voyles IT, Roy CJ, et al. Probability bounds analysis applied to the Sandia verification and validation challenge problem. Journal of Verification, Validation and Uncertainty Quantification. 2016; 1: 011003–011016.

[25]Li W, Chen S, Jiang Z, et al. Integrating baysian calibration, bias correction, and machine learning for the 2014 Sandia verification and validation challenge problem. Journal of Verification, Validation and Uncertainty Quantification. 2016; 1: 011004–011016.

[26]Xi Z, Yang RJ. Reliability analysis with model uncertainty coupling with parameter and experiment uncertainties: A case study of 2014 verification and validation challenge problem. Journal of Verification, Validation and Uncertainty Quantification. 2016; 1: 011005–011016.

[27]Mullins J, Mahadevan S. Bayesian uncertainty integration for model calibration, validation, and prediction. Journal of Verification, Validation and Uncertainty Quantification. 2016; 1: 011006–011016.

[28]Hazelrigg GA, Klutke GA. Models, uncertainty, and the Sandia V&V challenge problem. Journal of Verification, Validation and Uncertainty Quantification. 2020; 5: 015501–015507.

[29]Paez PJ, Paez TL, Hasselman, TK. Economic analysis of model validation for a challenge problem. Journal of Verification, Validation and Uncertainty Quantification. 2016; 1: 011007–011020.

[30]Hájek A. Scotching Dutch Books? Philosophical Perspectives. 2005; 19(1): 139–151.

[31]Hájek A. Dutch Book Arguments. In: Anand P, Pattanaik P, Puppe C (ediotors). The Oxford Handbook of Rational and Social Choice. Oxford University Press; 2009

[32]Caves CM. Probabilities as betting odds and the Dutch Book. Available online: http://info.phys.unm.edu/ caves/reports/dutchbook.pdf (accessed on 27 May 2025).

[33]Carroll L. Alice’s Adventures in Wonderland. United Kingdom; 1865.

[34]Hazelrigg GA. The Cheshire Cat on engineering design. Quality and Reliability Engineering International. 2009; 25: 759–769.

[35]von Neumann J, Morgenstern O. The Theory of Games and Economic Behavior, 3rd ed. Princeton University Press; 1953.

[36]Savage LJ. The Foundations of Statistics. Dover Publications, Inc.; 1972.

[37]Luce RD, Raiffa H. Games and Decisions. John Wiley & Sons; 1957.

[38]Bernardo JM, Smith AFM. Bayesian Theory. Wiley; 2000.

[39]Arrow KJ. Social Choice and Individual Values, 2nd ed. Wiley; 1963.

[40]Shannon CE. A mathematical theory of communication. The Bell System Technical Journal. 1984; 27: 379–423, 623–656.

[41]Lindley DV. Understanding Uncertainty. John Wiley & Sons; 2014.

[42]Lindley DV. Scoring rules and the inevitability of probability. International Statistical Review. 1982; 50: 1–26.

[43]Lindley DV. The philosophy of statistics. The Statistician. 2000; 49: 293–337.

[44]Ramsey FP. Truth and Probability. In: The Foundations of Mathematics and other Logical Essays, Ch. VII. Harcourt, Brace and Company; 1931. pp. 156–198.

[45]de Finetti B. Theory of Probability. Wiley; 1974. Volume 1.

[46]de Finetti B. 1975. Theory of Probability. Wiley; 1975; Volume 2.

[47]de Finetti B. Theory of Probability. John Wiley & Sons, Ltd.; 1990.

[48]DeGroot MH. Optimal Statistical Decisions. McGraw-Hill; 1970.

[49]Ferson S, Oberkampf WL, Ginzburg L. Validation of imprecise probability models. Sandia National Laboratories; 2008.

[50]Beer M, Ferson S, Kreinovich V. Imprecise probabilities in engineering analysis. University of Texas at El Paso; 2013.

[51]Bernoulli D. Exposition of a new theory on the measurement of risk. In: Page A (editor). Utility Theory: A Book of Readings. John Wiley & Sons, Inc.; 1968. pp. 199–214.

[52]Jolini S, Hazelrigg G.A. Differential utility: Accounting for correlation in performance among design alternatives. In: Proceedings of the ASME 2021 IDETC/CIE; August 17–19 2021.

[53]Hazelrigg GA, Busch AC. Setting navigational performance standards for aircraft flying over the oceans. Transportation Research Part A: General. 1986; 20(5): 373–383.