Continuous-Time Markov Decision Processes with Controlled Observations

Yunhan Huang, Veeraruna Kavitha, Quanyan Zhu

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

In this paper, we study a continuous-time discounted jump Markov decision process with both controlled actions and observations. The observation is only available for a discrete set of time instances. At each time of observation, one has to select an optimal timing for the next observation and a control trajectory for the time interval between two observation points. We provide a theoretical framework that the decision maker can utilize to find the optimal observation epochs and the optimal actions jointly. Two cases are investigated. One is gated queueing systems in which we explicitly characterize the optimal action and the optimal observation where the optimal observation is shown to be independent of the state. Another is the inventory control problem with Poisson arrival process in which we obtain numerically the optimal action and observation. The results show that it is optimal to observe more frequently at a region of states where the optimal action adapts constantly.

Original languageEnglish (US)
Title of host publication2019 57th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages32-39
Number of pages8
ISBN (Electronic)9781728131511
DOIs
StatePublished - Sep 2019
Event57th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2019 - Monticello, United States
Duration: Sep 24 2019Sep 27 2019

Publication series

Name2019 57th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2019

Conference

Conference57th Annual Allerton Conference on Communication, Control, and Computing, Allerton 2019
CountryUnited States
CityMonticello
Period9/24/199/27/19

Keywords

  • Controlled Observation
  • Dynamic Programming
  • Inventory Control
  • Markov Decision Process
  • Markov Jump Process
  • Queueing Systems
  • Value Iteration

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Networks and Communications
  • Hardware and Architecture
  • Safety, Risk, Reliability and Quality
  • Control and Optimization

Fingerprint Dive into the research topics of 'Continuous-Time Markov Decision Processes with Controlled Observations'. Together they form a unique fingerprint.

Cite this