Efficient nonlinear optimal smoothing and sampling algorithms for complex turbulent nonlinear dynamical systems with partial observations

A nonlinear optimal smoother and an associated optimal strategy of sampling hidden model trajectories are developed for a rich class of complex nonlinear turbulent dynamical systems with partial and noisy observations. Despite the strong nonlinearity and the significant non-Gaussian characteristics in the underlying systems, both the optimal smoother estimates and the sampled trajectories can be solved via closed analytic formulae. Thus, they are computationally efficient and the methods are applicable to high-dimensional systems. The nonlinear optimal smoother is able to estimate the hidden model states associated with various non-Gaussian phenomena and is particularly skillful in capturing the onset, demise and amplitude of the observed and hidden extreme events. On the other hand, the sampled hidden trajectories succeed in recovering both the dynamical and statistical features of the underlying nonlinear systems, including the fat-tailed non-Gaussian probability density function and the temporal autocorrelation function. In the situations with only a short period of partially observed training time series, the optimal sampling strategy can be used to efficiently create a sufficient number of samples in an unbiased fashion that facilitates an accurate prediction of important non-Gaussian features in both the observed and hidden variables. In addition, the information provided by the sampled trajectories based on imperfect models allows an effective way of quantifying the model error. It also offers a systematic approach to improve approximate models and stochastic parameterizations in highly non-Gaussian systems and thus advances the real-time forecasts.