Adaptive Honeypot Engagement Through Reinforcement Learning of Semi-Markov Decision Processes

Linan Huang, Quanyan Zhu

Research output: Chapter in Book/Report/Conference proceedingConference contribution


A honeynet is a promising active cyber defense mechanism. It reveals the fundamental Indicators of Compromise (IoCs) by luring attackers to conduct adversarial behaviors in a controlled and monitored environment. The active interaction at the honeynet brings a high reward but also introduces high implementation costs and risks of adversarial honeynet exploitation. In this work, we apply infinite-horizon Semi-Markov Decision Process (SMDP) to characterize a stochastic transition and sojourn time of attackers in the honeynet and quantify the reward-risk trade-off. In particular, we design adaptive long-term engagement policies shown to be risk-averse, cost-effective, and time-efficient. Numerical results have demonstrated that our adaptive engagement policies can quickly attract attackers to the target honeypot and engage them for a sufficiently long period to obtain worthy threat information. Meanwhile, the penetration probability is kept at a low level. The results show that the expected utility is robust against attackers of a large range of persistence and intelligence. Finally, we apply reinforcement learning to the SMDP to solve the curse of modeling. Under a prudent choice of the learning rate and exploration policy, we achieve a quick and robust convergence of the optimal policy and value.

Original languageEnglish (US)
Title of host publicationDecision and Game Theory for Security - 10th International Conference, GameSec 2019, Proceedings
EditorsTansu Alpcan, Yevgeniy Vorobeychik, John S. Baras, György Dán
Number of pages21
ISBN (Print)9783030324292
StatePublished - 2019
Event10th International Conference on Decision and Game Theory for Security, GameSec 2019 - Stockholm, Sweden
Duration: Oct 30 2019Nov 1 2019

Publication series

NameLecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)
Volume11836 LNCS
ISSN (Print)0302-9743
ISSN (Electronic)1611-3349


Conference10th International Conference on Decision and Game Theory for Security, GameSec 2019


  • Active defense
  • Honeynet
  • Reinforcement learning
  • Risk quantification
  • Semi-Markov decision processes

ASJC Scopus subject areas

  • Theoretical Computer Science
  • General Computer Science


Dive into the research topics of 'Adaptive Honeypot Engagement Through Reinforcement Learning of Semi-Markov Decision Processes'. Together they form a unique fingerprint.

Cite this