Addressing Sample Efficiency and Model-bias in Model-based Reinforcement Learning

Akhil S. Anand; Jens Erik Kveen; Fares Abu-Dakka; Esten Ingar Grøtli; Jan Tommy Gravdahl

doi:10.1109/ICMLA55696.2022.00009

Addressing Sample Efficiency and Model-bias in Model-based Reinforcement Learning

Akhil S. Anand, Jens Erik Kveen, Fares Abu-Dakka, Esten Ingar Grøtli, Jan Tommy Gravdahl

Mechanical Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

Model-based reinforcement learning promises to be an effective way to bring reinforcement learning to real-world robotic systems by offering a sample efficient learning approach compared to model-free reinforcement learning. However, model-based reinforcement learning approaches at present struggle to match the performance of model-free ones. This work attempts to fill this gap by improving the performance of model-based reinforcement learning while further improving its sample efficiency. To improve the sample efficiency, an exploration strategy is formulated which maximizes the information gain. The asymptotic performance is improved by compensating for the model-bias using a model-free critic. We have evaluated our proposed approach on four reinforcement learning benchmarking tasks in the openAI gym framework.

Original language	English (US)
Title of host publication	Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022
Editors	M. Arif Wani, Mehmed Kantardzic, Vasile Palade, Daniel Neagu, Longzhi Yang, Kit-Yan Chan
Publisher	Institute of Electrical and Electronics Engineers Inc.
Pages	1-6
Number of pages	6
ISBN (Electronic)	9781665462839
DOIs	https://doi.org/10.1109/ICMLA55696.2022.00009
State	Published - 2022
Event	21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022 - Nassau, Bahamas Duration: Dec 12 2022 → Dec 14 2022

Publication series

Name	Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022

Conference

Conference	21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022
Country/Territory	Bahamas
City	Nassau
Period	12/12/22 → 12/14/22

Keywords

Model based reinforcement learning
model predictive control
sample efficient learning

ASJC Scopus subject areas

Computer Vision and Pattern Recognition
Computer Science Applications
Artificial Intelligence
Hardware and Architecture

Access to Document

10.1109/ICMLA55696.2022.00009

Cite this

Anand, A. S., Erik Kveen, J., Abu-Dakka, F., Grøtli, E. I., & Tommy Gravdahl, J. (2022). Addressing Sample Efficiency and Model-bias in Model-based Reinforcement Learning. In M. A. Wani, M. Kantardzic, V. Palade, D. Neagu, L. Yang, & K.-Y. Chan (Eds.), Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022 (pp. 1-6). (Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICMLA55696.2022.00009

Addressing Sample Efficiency and Model-bias in Model-based Reinforcement Learning. / Anand, Akhil S.; Erik Kveen, Jens; Abu-Dakka, Fares et al.
Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022. ed. / M. Arif Wani; Mehmed Kantardzic; Vasile Palade; Daniel Neagu; Longzhi Yang; Kit-Yan Chan. Institute of Electrical and Electronics Engineers Inc., 2022. p. 1-6 (Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Anand, AS, Erik Kveen, J, Abu-Dakka, F, Grøtli, EI & Tommy Gravdahl, J 2022, Addressing Sample Efficiency and Model-bias in Model-based Reinforcement Learning. in MA Wani, M Kantardzic, V Palade, D Neagu, L Yang & K-Y Chan (eds), Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022. Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022, Institute of Electrical and Electronics Engineers Inc., pp. 1-6, 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022, Nassau, Bahamas, 12/12/22. https://doi.org/10.1109/ICMLA55696.2022.00009

Anand AS, Erik Kveen J, Abu-Dakka F, Grøtli EI, Tommy Gravdahl J. Addressing Sample Efficiency and Model-bias in Model-based Reinforcement Learning. In Wani MA, Kantardzic M, Palade V, Neagu D, Yang L, Chan KY, editors, Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022. Institute of Electrical and Electronics Engineers Inc. 2022. p. 1-6. (Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022). doi: 10.1109/ICMLA55696.2022.00009

Anand, Akhil S. ; Erik Kveen, Jens ; Abu-Dakka, Fares et al. / Addressing Sample Efficiency and Model-bias in Model-based Reinforcement Learning. Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022. editor / M. Arif Wani ; Mehmed Kantardzic ; Vasile Palade ; Daniel Neagu ; Longzhi Yang ; Kit-Yan Chan. Institute of Electrical and Electronics Engineers Inc., 2022. pp. 1-6 (Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022).

@inproceedings{c58761902d12441e8167142794cc0ed3,

title = "Addressing Sample Efficiency and Model-bias in Model-based Reinforcement Learning",

abstract = "Model-based reinforcement learning promises to be an effective way to bring reinforcement learning to real-world robotic systems by offering a sample efficient learning approach compared to model-free reinforcement learning. However, model-based reinforcement learning approaches at present struggle to match the performance of model-free ones. This work attempts to fill this gap by improving the performance of model-based reinforcement learning while further improving its sample efficiency. To improve the sample efficiency, an exploration strategy is formulated which maximizes the information gain. The asymptotic performance is improved by compensating for the model-bias using a model-free critic. We have evaluated our proposed approach on four reinforcement learning benchmarking tasks in the openAI gym framework.",

keywords = "Model based reinforcement learning, model predictive control, sample efficient learning",

author = "Anand, {Akhil S.} and {Erik Kveen}, Jens and Fares Abu-Dakka and Gr{\o}tli, {Esten Ingar} and {Tommy Gravdahl}, Jan",

note = "Publisher Copyright: {\textcopyright} 2022 IEEE.; 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022 ; Conference date: 12-12-2022 Through 14-12-2022",

year = "2022",

doi = "10.1109/ICMLA55696.2022.00009",

language = "English (US)",

series = "Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

pages = "1--6",

editor = "Wani, {M. Arif} and Mehmed Kantardzic and Vasile Palade and Daniel Neagu and Longzhi Yang and Kit-Yan Chan",

booktitle = "Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022",

}

TY - GEN

T1 - Addressing Sample Efficiency and Model-bias in Model-based Reinforcement Learning

AU - Anand, Akhil S.

AU - Erik Kveen, Jens

AU - Abu-Dakka, Fares

AU - Grøtli, Esten Ingar

AU - Tommy Gravdahl, Jan

PY - 2022

Y1 - 2022

N2 - Model-based reinforcement learning promises to be an effective way to bring reinforcement learning to real-world robotic systems by offering a sample efficient learning approach compared to model-free reinforcement learning. However, model-based reinforcement learning approaches at present struggle to match the performance of model-free ones. This work attempts to fill this gap by improving the performance of model-based reinforcement learning while further improving its sample efficiency. To improve the sample efficiency, an exploration strategy is formulated which maximizes the information gain. The asymptotic performance is improved by compensating for the model-bias using a model-free critic. We have evaluated our proposed approach on four reinforcement learning benchmarking tasks in the openAI gym framework.

AB - Model-based reinforcement learning promises to be an effective way to bring reinforcement learning to real-world robotic systems by offering a sample efficient learning approach compared to model-free reinforcement learning. However, model-based reinforcement learning approaches at present struggle to match the performance of model-free ones. This work attempts to fill this gap by improving the performance of model-based reinforcement learning while further improving its sample efficiency. To improve the sample efficiency, an exploration strategy is formulated which maximizes the information gain. The asymptotic performance is improved by compensating for the model-bias using a model-free critic. We have evaluated our proposed approach on four reinforcement learning benchmarking tasks in the openAI gym framework.

KW - Model based reinforcement learning

KW - model predictive control

KW - sample efficient learning

UR - http://www.scopus.com/inward/record.url?scp=85152213482&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85152213482&partnerID=8YFLogxK

U2 - 10.1109/ICMLA55696.2022.00009

DO - 10.1109/ICMLA55696.2022.00009

M3 - Conference contribution

AN - SCOPUS:85152213482

T3 - Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022

SP - 1

EP - 6

BT - Proceedings - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022

A2 - Wani, M. Arif

A2 - Kantardzic, Mehmed

A2 - Palade, Vasile

A2 - Neagu, Daniel

A2 - Yang, Longzhi

A2 - Chan, Kit-Yan

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - 21st IEEE International Conference on Machine Learning and Applications, ICMLA 2022

Y2 - 12 December 2022 through 14 December 2022

ER -

Addressing Sample Efficiency and Model-bias in Model-based Reinforcement Learning

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this