ERA: Entity-Relationship Aware Video Summarization with Wasserstein GAN

Guande Wu, Jianzhe Lin, Claudio T. Silva

Research output: Contribution to conferencePaperpeer-review


Video summarization aims to simplify large-scale video browsing by generating concise, short summaries that diver from but well represent the original video. Due to the scarcity of video annotations, recent progress for video summarization concentrates on unsupervised methods, among which the GAN-based methods are most prevalent. This type of methods includes a summarizer and a discriminator. The summarized video from the summarizer will be assumed as the final output, only if the video reconstructed from this summary cannot be discriminated from the original one by the discriminator. The primary problems of this GAN-based methods are two-folds. First, the summarized video in this way is a subset of original video with low redundancy and contains high priority events/entities. This summarization criterion is not enough. Second, the training of the GAN framework is not stable. This paper proposes a novel Entity-relationship Aware video summarization method (ERA) to address the above problems. To be more specific, we introduce an Adversarial Spatio-Temporal network to construct the relationship among entities, which we think should also be given high priority in the summarization. The GAN training problem is solved by introducing the Wasserstein GAN and two newly proposed video-patch/score-sum losses. In addition, the score-sum loss can also relieve the model sensitivity to the varying video lengths, which is an inherent problem for most current video analysis tasks. Our method substantially lifts the performance on the target benchmark datasets and exceeds the current state-of-the-art. We hope our straightforward yet effective approach will shed some light on the future research of unsupervised video summarization. The code is available online.

Original languageEnglish (US)
StatePublished - 2021
Event32nd British Machine Vision Conference, BMVC 2021 - Virtual, Online
Duration: Nov 22 2021Nov 25 2021


Conference32nd British Machine Vision Conference, BMVC 2021
CityVirtual, Online

ASJC Scopus subject areas

  • Artificial Intelligence
  • Computer Vision and Pattern Recognition


Dive into the research topics of 'ERA: Entity-Relationship Aware Video Summarization with Wasserstein GAN'. Together they form a unique fingerprint.

Cite this