On Convergence Rate of Adaptive Multiscale Value Function Approximation for Reinforcement Learning

Tao Li; Quanyan Zhu

doi:10.1109/MLSP.2019.8918816

On Convergence Rate of Adaptive Multiscale Value Function Approximation for Reinforcement Learning

Tao Li, Quanyan Zhu

Electrical and Computer Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In this paper, we propose a generic framework for devising an adaptive approximation scheme for value function approximation in reinforcement learning, which introduces multiscale approximation. The two basic ingredients are multiresolution analysis as well as tree approximation. Starting from simple refinable functions, multiresolution analysis enables us to construct a wavelet system from which the basis functions are selected adaptively, resulting in a tree structure. Furthermore, we present the convergence rate of our multiscale approximation which does not depend on the regularity of basis functions.

Original language	English (US)
Title of host publication	2019 IEEE 29th International Workshop on Machine Learning for Signal Processing, MLSP 2019
Publisher	IEEE Computer Society
ISBN (Electronic)	9781728108247
DOIs	https://doi.org/10.1109/MLSP.2019.8918816
State	Published - Oct 2019
Event	29th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2019 - Pittsburgh, United States Duration: Oct 13 2019 → Oct 16 2019

Publication series

Name	IEEE International Workshop on Machine Learning for Signal Processing, MLSP
Volume	2019-October
ISSN (Print)	2161-0363
ISSN (Electronic)	2161-0371

Conference

Conference	29th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2019
Country/Territory	United States
City	Pittsburgh
Period	10/13/19 → 10/16/19

Keywords

Multiscale approximation
multiresolution analysis
n-term approximation
reinforcement learning
tree approximation
wavelets

ASJC Scopus subject areas

Human-Computer Interaction
Signal Processing

Access to Document

10.1109/MLSP.2019.8918816

Cite this

Li, T., & Zhu, Q. (2019). On Convergence Rate of Adaptive Multiscale Value Function Approximation for Reinforcement Learning. In 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing, MLSP 2019 Article 8918816 (IEEE International Workshop on Machine Learning for Signal Processing, MLSP; Vol. 2019-October). IEEE Computer Society. https://doi.org/10.1109/MLSP.2019.8918816

On Convergence Rate of Adaptive Multiscale Value Function Approximation for Reinforcement Learning. / Li, Tao; Zhu, Quanyan.
2019 IEEE 29th International Workshop on Machine Learning for Signal Processing, MLSP 2019. IEEE Computer Society, 2019. 8918816 (IEEE International Workshop on Machine Learning for Signal Processing, MLSP; Vol. 2019-October).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Li, T & Zhu, Q 2019, On Convergence Rate of Adaptive Multiscale Value Function Approximation for Reinforcement Learning. in 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing, MLSP 2019., 8918816, IEEE International Workshop on Machine Learning for Signal Processing, MLSP, vol. 2019-October, IEEE Computer Society, 29th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2019, Pittsburgh, United States, 10/13/19. https://doi.org/10.1109/MLSP.2019.8918816

@inproceedings{0224645e786b407296bbf48244cccda9,

title = "On Convergence Rate of Adaptive Multiscale Value Function Approximation for Reinforcement Learning",

abstract = "In this paper, we propose a generic framework for devising an adaptive approximation scheme for value function approximation in reinforcement learning, which introduces multiscale approximation. The two basic ingredients are multiresolution analysis as well as tree approximation. Starting from simple refinable functions, multiresolution analysis enables us to construct a wavelet system from which the basis functions are selected adaptively, resulting in a tree structure. Furthermore, we present the convergence rate of our multiscale approximation which does not depend on the regularity of basis functions.",

keywords = "Multiscale approximation, multiresolution analysis, n-term approximation, reinforcement learning, tree approximation, wavelets",

author = "Tao Li and Quanyan Zhu",

note = "Funding Information: This research is supported in part by National Science Foundation (NSF) under grant ECCS-1847056, CNS-1544782, and SES-1541164, and in part by ARO grant W911NF1910041. Publisher Copyright: {\textcopyright} 2019 IEEE.; 29th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2019 ; Conference date: 13-10-2019 Through 16-10-2019",

year = "2019",

month = oct,

doi = "10.1109/MLSP.2019.8918816",

language = "English (US)",

series = "IEEE International Workshop on Machine Learning for Signal Processing, MLSP",

publisher = "IEEE Computer Society",

booktitle = "2019 IEEE 29th International Workshop on Machine Learning for Signal Processing, MLSP 2019",

}

TY - GEN

T1 - On Convergence Rate of Adaptive Multiscale Value Function Approximation for Reinforcement Learning

AU - Li, Tao

AU - Zhu, Quanyan

N1 - Funding Information: This research is supported in part by National Science Foundation (NSF) under grant ECCS-1847056, CNS-1544782, and SES-1541164, and in part by ARO grant W911NF1910041. Publisher Copyright: © 2019 IEEE.

PY - 2019/10

Y1 - 2019/10

N2 - In this paper, we propose a generic framework for devising an adaptive approximation scheme for value function approximation in reinforcement learning, which introduces multiscale approximation. The two basic ingredients are multiresolution analysis as well as tree approximation. Starting from simple refinable functions, multiresolution analysis enables us to construct a wavelet system from which the basis functions are selected adaptively, resulting in a tree structure. Furthermore, we present the convergence rate of our multiscale approximation which does not depend on the regularity of basis functions.

AB - In this paper, we propose a generic framework for devising an adaptive approximation scheme for value function approximation in reinforcement learning, which introduces multiscale approximation. The two basic ingredients are multiresolution analysis as well as tree approximation. Starting from simple refinable functions, multiresolution analysis enables us to construct a wavelet system from which the basis functions are selected adaptively, resulting in a tree structure. Furthermore, we present the convergence rate of our multiscale approximation which does not depend on the regularity of basis functions.

KW - Multiscale approximation

KW - multiresolution analysis

KW - n-term approximation

KW - reinforcement learning

KW - tree approximation

KW - wavelets

UR - http://www.scopus.com/inward/record.url?scp=85077700076&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85077700076&partnerID=8YFLogxK

U2 - 10.1109/MLSP.2019.8918816

DO - 10.1109/MLSP.2019.8918816

M3 - Conference contribution

AN - SCOPUS:85077700076

T3 - IEEE International Workshop on Machine Learning for Signal Processing, MLSP

BT - 2019 IEEE 29th International Workshop on Machine Learning for Signal Processing, MLSP 2019

PB - IEEE Computer Society

T2 - 29th IEEE International Workshop on Machine Learning for Signal Processing, MLSP 2019

Y2 - 13 October 2019 through 16 October 2019

ER -

On Convergence Rate of Adaptive Multiscale Value Function Approximation for Reinforcement Learning

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this