Field-of-view prediction in 360-degree videos with attention-based neural encoder-decoder networks

Jiang Yu; Yong Liu

doi:10.1145/3304113.3326118

Field-of-view prediction in 360-degree videos with attention-based neural encoder-decoder networks

Jiang Yu, Yong Liu

Electrical and Computer Engineering

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Abstract

In this paper, we propose attention-based neural encoder-decoder networks for predicting user Field-of-View (FoV) in 360-degree videos. Our proposed prediction methods are based on the attention mechanism that learns the weighted prediction power of historical FoV time series through end-to-end training. Attention-based neural encoder-decoder networks do not involve recursion, thus can be highly parallelized during training. Using publicly available 360-degree head movement datasets, we demonstrate that our FoV prediction models outperform the state-of-art FoV prediction models, achieving lower prediction error, higher training throughput, and faster convergence. Better FoV prediction leads to reduced bandwidth consumption, better video quality, and improved user quality of experience.

Original language	English (US)
Title of host publication	Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019
Publisher	Association for Computing Machinery, Inc
Pages	37-42
Number of pages	6
ISBN (Electronic)	9781450362993
DOIs	https://doi.org/10.1145/3304113.3326118
State	Published - Jun 18 2019
Event	11th ACM SIGMM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019 - Amherst, United States Duration: Jun 18 2019 → …

Publication series

Name	Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019

Conference

Conference	11th ACM SIGMM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019
Country/Territory	United States
City	Amherst
Period	6/18/19 → …

Keywords

360 degree videos
Attention
Encoder decoder networks
Field of view prediction
Neural networks

ASJC Scopus subject areas

Human-Computer Interaction
Software
Computer Graphics and Computer-Aided Design

Access to Document

10.1145/3304113.3326118

Cite this

Yu, J., & Liu, Y. (2019). Field-of-view prediction in 360-degree videos with attention-based neural encoder-decoder networks. In Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019 (pp. 37-42). (Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019). Association for Computing Machinery, Inc. https://doi.org/10.1145/3304113.3326118

Field-of-view prediction in 360-degree videos with attention-based neural encoder-decoder networks. / Yu, Jiang; Liu, Yong.
Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019. Association for Computing Machinery, Inc, 2019. p. 37-42 (Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

Yu, J & Liu, Y 2019, Field-of-view prediction in 360-degree videos with attention-based neural encoder-decoder networks. in Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019. Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019, Association for Computing Machinery, Inc, pp. 37-42, 11th ACM SIGMM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019, Amherst, United States, 6/18/19. https://doi.org/10.1145/3304113.3326118

Yu J, Liu Y. Field-of-view prediction in 360-degree videos with attention-based neural encoder-decoder networks. In Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019. Association for Computing Machinery, Inc. 2019. p. 37-42. (Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019). doi: 10.1145/3304113.3326118

Yu, Jiang ; Liu, Yong. / Field-of-view prediction in 360-degree videos with attention-based neural encoder-decoder networks. Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019. Association for Computing Machinery, Inc, 2019. pp. 37-42 (Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019).

@inproceedings{62f9adaf66644c91b8f938af67141f32,

title = "Field-of-view prediction in 360-degree videos with attention-based neural encoder-decoder networks",

abstract = "In this paper, we propose attention-based neural encoder-decoder networks for predicting user Field-of-View (FoV) in 360-degree videos. Our proposed prediction methods are based on the attention mechanism that learns the weighted prediction power of historical FoV time series through end-to-end training. Attention-based neural encoder-decoder networks do not involve recursion, thus can be highly parallelized during training. Using publicly available 360-degree head movement datasets, we demonstrate that our FoV prediction models outperform the state-of-art FoV prediction models, achieving lower prediction error, higher training throughput, and faster convergence. Better FoV prediction leads to reduced bandwidth consumption, better video quality, and improved user quality of experience.",

keywords = "360 degree videos, Attention, Encoder decoder networks, Field of view prediction, Neural networks",

author = "Jiang Yu and Yong Liu",

note = "Publisher Copyright: {\textcopyright} 2019 Association for Computing Machinery.; 11th ACM SIGMM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019 ; Conference date: 18-06-2019",

year = "2019",

month = jun,

day = "18",

doi = "10.1145/3304113.3326118",

language = "English (US)",

series = "Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019",

publisher = "Association for Computing Machinery, Inc",

pages = "37--42",

booktitle = "Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019",

}

TY - GEN

T1 - Field-of-view prediction in 360-degree videos with attention-based neural encoder-decoder networks

AU - Yu, Jiang

AU - Liu, Yong

PY - 2019/6/18

Y1 - 2019/6/18

N2 - In this paper, we propose attention-based neural encoder-decoder networks for predicting user Field-of-View (FoV) in 360-degree videos. Our proposed prediction methods are based on the attention mechanism that learns the weighted prediction power of historical FoV time series through end-to-end training. Attention-based neural encoder-decoder networks do not involve recursion, thus can be highly parallelized during training. Using publicly available 360-degree head movement datasets, we demonstrate that our FoV prediction models outperform the state-of-art FoV prediction models, achieving lower prediction error, higher training throughput, and faster convergence. Better FoV prediction leads to reduced bandwidth consumption, better video quality, and improved user quality of experience.

AB - In this paper, we propose attention-based neural encoder-decoder networks for predicting user Field-of-View (FoV) in 360-degree videos. Our proposed prediction methods are based on the attention mechanism that learns the weighted prediction power of historical FoV time series through end-to-end training. Attention-based neural encoder-decoder networks do not involve recursion, thus can be highly parallelized during training. Using publicly available 360-degree head movement datasets, we demonstrate that our FoV prediction models outperform the state-of-art FoV prediction models, achieving lower prediction error, higher training throughput, and faster convergence. Better FoV prediction leads to reduced bandwidth consumption, better video quality, and improved user quality of experience.

KW - 360 degree videos

KW - Attention

KW - Encoder decoder networks

KW - Field of view prediction

KW - Neural networks

UR - http://www.scopus.com/inward/record.url?scp=85069473778&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=85069473778&partnerID=8YFLogxK

U2 - 10.1145/3304113.3326118

DO - 10.1145/3304113.3326118

M3 - Conference contribution

AN - SCOPUS:85069473778

T3 - Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019

SP - 37

EP - 42

BT - Proceedings of the 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019

PB - Association for Computing Machinery, Inc

T2 - 11th ACM SIGMM Workshop on Immersive Mixed and Virtual Environment Systems, MMVE 2019

Y2 - 18 June 2019

ER -

Field-of-view prediction in 360-degree videos with attention-based neural encoder-decoder networks

Abstract

Publication series

Conference

Keywords

ASJC Scopus subject areas

Access to Document

Other files and links

Fingerprint

Cite this