MUCM
 

MUCM

 

 

ABOUT MUCM

The MUCM (Managing Uncertainty in Complex Models) project, funded by Research Councils UK, is a collaboration between five universities: Sheffield, Aston, Southampton, Durham, and the London School of Economics and incorporates advisors from across UK and Europe as well as the USA.

About Uncertainty in Models
Background to MUCM
Plans for MUCM2



ABOUT UNCERTAINTY IN MODELS

Modelling is a vital part of research and development in almost every sphere of modern life, as a few examples will suffice to demonstrate:
- Climate models, incorporating very complex equations of atmospheric and sea surface processes (and even more complex models that widen the scope to include full ocean modelling) are used for weather and climate forecasting. Some of the largest computers in the world are used to run weather forecasting models.  Running such models to forecast long-term climate change and its impacts is even more computationally demanding, yet hugely important for national and international policy-making.
- In order to predict the behaviour of nuclear power reactors, nuclear waste storage facilities and high-energy physics experiments, very complex models are used incorporating the latest nuclear physics theory.  The safety of such installations depends in part on the accuracy of these models, and the models are an integral part of their monitoring and regulation.
- In developing large engineering projects, it is standard practice to build a theoretical model of the proposed equipment in order to predict its behaviour and to set the design parameters to obtain optimal results.  This avoids the need to make many expensive prototypes, and is used for everything from car engines to aircraft wings to the hulls of ocean racing yachts.

Those who rely on models to understand complex processes, and for prediction, optimisation and many kinds of decision-and policy-making, increasingly wish to know how much they can trust the model outputs.  Uncertainty and inaccuracy in the outputs arises from numerous sources, including error in initial conditions, error in model parameters, imperfect science in the model equations, approximate solutions to model equations and errors in model structure or logic.  The nature and magnitudes of these contributory uncertainties are often very difficult to estimate, but it is vital to do so.  All the while, for instance, different models produce very different predictions of the magnitude of global warming effects, with no credible error bounds, sceptics can continue to ignore them and pressure groups will seize upon the most pessimistic predictions.

The MUCM project will develop a technology that is capable of addressing all sources of uncertainty in model predictions and to quantify their implications efficiently, even in the most complex models.  It has the potential to revolutionise scientific debate by resolving the contradictions in competing models.  It will also have a radical effect on everyday modelling and model usage by making the uncertainties in model outputs transparent to modellers and end users alike.


In particular, this technology will provide a unified framework in which to address the following tasks that frequently arise in the use of models.

Uncertainty Analysis

This is the task of quantifying the overall uncertainty in model outputs.  In principle, this involves taking account of all sources of uncertainty, although it has often in the past been interpreted more narrowly as quantifying the uncertainty in outputs due to input uncertainty.

Sensitivity Analysis and value of information

A related task is that of identifying how the model output responds to individual inputs (or other uncertain factors).  Variance-based sensitivity analysis summarises the sensitivity by identifying how much of the output variance is due to each contributory source of uncertainty. Value of information analysis identifies how much each source of uncertainty impacts on decision-making. These tools give modellers insight into how their models behave (often pointing to bugs in coding or model failures) and allow model users to prioritise research to reduce uncertainty.

Calibration and data assimilation

Calibration, the process of adjusting uncertain model parameters to fit the model to observed data, is typically a very demanding task that can involve many man months or even years of effort. Data assimilation, in which data are used to adjust the state vector of a dynamic model, is equally demanding and the subject of quite intensive research in its own right. Gaussian process methods can not only perform these tasks more efficiently, but also properly characterise how they reduce uncertainty about those parameters and state vector (and hence reduce uncertainty in model outputs).

Validation and structural uncertainty

It is often said that models cannot be validated since no model is perfect. Nevertheless, it is possible to validate the combination of a model with a description of uncertainty, simply by computing implied probability distributions for test data and then verifying that they lie within the bounds of those distributions. However, this requires all forms of uncertainty to be accounted for, including uncertainty in model structure, and cannot be addressed by conventional Monte Carlo analyses. Bayesian statistical methods are able to tackle this problem, and indeed a model for model discrepancy underlies the calibration techniques


BACKGROUND TO MUCM

Modelling is a vital part of research and development in almost every sphere of modern life. Those who rely on models to understand complex processes, and for prediction, optimisation and many kinds of decision- and policy-making, increasingly wish to know how much they can trust the model outputs. Uncertainty and inaccuracy in the outputs arises from numerous sources, including error in initial conditions, error in model parameters, imperfect science in the model equations, approximate solutions to model equations and errors in model structure or logic. The nature and magnitudes of these contributory uncertainties are often very difficult to estimate, but it is vital to do so. All the while, for instance, different models produce very different predictions of the magnitude of global warming effects, with no credible error bounds, sceptics can continue to ignore them and pressure groups will seize upon the most pessimistic predictions.

The focus of the MUCM project has been to develop a technology that is capable of addressing all sources of uncertainty in model predictions and to quantify their implications efficiently, even in highly complex models. Furthermore, MUCM seeks to facilitate the management of uncertainty through tools to show how much of the overall uncertainty is due to each contributing source of uncertainty (sensitivity analysis) and to reduce uncertainty by learning from observations of the real-world process (calibration and data assimilation).

MUCM methods are based on two fundamental technical developments. The first is the creation of an emulator for the computer model (usually called the simulator), using a Gaussian process or analogous Bayes linear theory. Emulation based methods are generally orders of magnitude more efficient than traditional Monte Carlo, requiring typically just a few hundreds of model runs, thereby providing very significant productivity gains for the researchers or analysis teams involved. The second development is to model the difference between the simulator output (using ‘best’ input values) and reality, which we call model discrepancy. This is a crucial step in making use of real-world observations, and with emulation allows an integrated approach to uncertainty analysis, sensitivity analysis, calibration and data assimilation – yet model discrepancy is not even acknowledged in traditional approaches to these tasks.

Overall, the project is on track to achieve its principal objectives – delivery of the MUCM technology via the toolkit and case studies, plus developments that explore and push back its boundaries of application.

There is growing international awareness of MUCM and the technology of emulating complex models including collaboration with SAMSI (the Statistics and Applied Mathematics Institute in the Research Triangle, North Carolina), and sessions devoted to this technology at several major international conferences such as the World Congress in Probability and Statistics, the International Society for Bayesian Analysis World Meeting and the International Statistical Institute Annual Meeting.

PLANS FOR MUCM2

We have been awarded a two year extension to the MUCM project (MUCM2). Within MUCM2, in addition to scoping three new areas of research, we plan to consolidate the work we have already done on disseminating our work to the user community. Our plans therefore have four components as described below.

Random Outputs
We will study several different kinds of random simulators with a view to scoping promising approaches to tackling the difficulties they post, and particularly to identifying the extent to which different classes of simulators can be addressed in common ways.   We will explore techniques for numerous problems in connection with each of the four classes of random simulators as appropriate (micro-simulation, agent models, systems biology models and SFEM).

Informing Decisions
Models are of course frequently used as an aid to making decisions or policies. In engineering, a simulator of a complex structure such as a motor car engine or a nuclear reactor is used to predict performance of a range of designs, and to select the design with optimal properties. In climate, policy on tackling climate change is necessarily based on models for the consequences of different emissions scenarios. This is another area in which the issues have turned out to be much more complex than we had supposed when we started the MUCM project so in MUCM2 we will address more fully the ways in which models can inform decisions, covering a number of topics including optimisation, coupling models, decisions that expand models, fitness for purpose and risk metrics.

Heterogeneity

The emulator methods developed in MUCM are predicated on the underlying simulator behaving more or less homogeneously across the input space. That is, the prior belief is that the simulator output should not respond much more dynamically to changes in a given input over some parts of the input space than over others. In particular, there is no prior expectation of sudden shifts, and certainly not discontinuities in the output as the inputs vary smoothly. Emulators can adapt to failure of these prior expectations, but may need large numbers of well-targeted training runs in order to do so. Validation diagnostics are likely to show poor validation even after fitting with a substantial training dataset has seemed to produce accurate emulation, because the predictive uncertainties still do not match appropriately with new validation model runs. Experience in MUCM, particularly with validation but also with some models having clearly heterogeneous behaviour, suggests that this is more of a problem than had been anticipated so we therefore plan to explore ideas for new kinds of emulator which can allow for prior expectations of heterogeneity including discontinuities and heterogeneous variance.

Building the Community

A strategic part of the original MUCM project has been to develop a toolkit and case studies to provide a resource for the community of model users and those working in the field. The toolkit was highly commended by our Mid Term Review panel. We have taken to heart their advice that the toolkit needs a strong editorial control to achieve consistency and quality, and O’Hagan has taken on the role of general toolkit editor. In MUCM2 we plan to further develop the toolkit and case studies, to extend the range of services to the community and to reach out to new user groups. Our objective is to build a real community of users and researchers that will be sustainable beyond the Translation award.  We intend to extend the toolkit, deliver short courses, organize video podcasts/seminars and develop the website further.Further information and webpages for MUCM2 will be made available as this year progresses, so please keep checking the website for new information