A new efficient parameter estimation algorithm for high-dimensional complex nonlinear turbulent dynamical systems with partial observations

Parameter estimation for high-dimensional complex nonlinear turbulent dynamical systems with only partial observations is an important and practical issue. However, most of the existing parameter estimation algorithms are computationally expensive in the presence of a large number of state variables or parameters. In this article, a parameter estimation algorithm is developed for high-dimensional nonlinear turbulent dynamical systems with conditional Gaussian structures. This algorithm exploits the closed analytical form of the conditional statistics to recover the unobserved trajectories in an optimal and deterministic way, which facilitates the calculation of the likelihood function and circumvents the computationally expensive data augmentation approach in sampling the unobserved trajectories as widely used in the literature. Such an efficient method of recovering the unobserved trajectories is then incorporated into a standard Markov chain Monte Carlo (MCMC) algorithm to estimate parameters in complex dynamical system using only a short period of training data. Next, in light of the dynamical features, two effective strategies are developed and incorporated into the algorithm that facilitates the parameter estimation of many high-dimensional systems. The first strategy involves a judicious block decomposition of the state variables such that the original problem is divided into several subproblems coupled in a specific way that allows an extremely cheap parallel computation for the parameter estimation. The second strategy exploits statistical symmetry for a further reduction of the computational cost when the system is statistically homogeneous. The new parameter estimation algorithm is applied to a two-layer Lorenz 96 model with 80 state variables and 162 parameters and the model mimics the realistic features of atmosphere wave propagations and excitable media. The efficient algorithm results in an accurate estimation of the parameters, which further allows a skillful prediction by the model with estimated parameters. Other simple nonlinear models are also used to illustrate the features of the new algorithm.