Introduction to Partially Observed MDPs

Quanyan Zhu, Zhiheng Xu

Research output: Chapter in Book/Report/Conference proceedingChapter


In our cross-layer design, we use different models to capture the properties of different layers. As stated in Chap. 8, we can use an MDP model to capture the dynamical movements of the cyber layer. However, in the scenario, we assume that the defender can observe the cyber state at each cyber time instant. In real applications, it is challenging to obtain the full information of the cyber state directly. Hence, the MDP cannot capture the incomplete knowledge of the cyber states. In this chapter, we will introduce a Partially Observed Markov Decision Process (POMDP) to capture the uncertainty of the cyber state. In a POMDP, instead of observing the states, we have an observation, whose distribution depends on the state. Therefore, we use this information to build a Hidden Markov Model (HMM) filter, which can construct a belief of the states. Based on the belief, we aim to find an optimal policy to minimize an expected cost.

Original languageEnglish (US)
Title of host publicationAdvances in Information Security
Number of pages7
StatePublished - 2020

Publication series

NameAdvances in Information Security
ISSN (Print)1568-2633

ASJC Scopus subject areas

  • Information Systems
  • Computer Networks and Communications


Dive into the research topics of 'Introduction to Partially Observed MDPs'. Together they form a unique fingerprint.

Cite this