Exact recovery in the stochastic block model

Emmanuel Abbe, Afonso S. Bandeira, Georgina Hall

Research output: Contribution to journalArticle

Abstract

The stochastic block model with two communities, or equivalently the planted bisection model, is a popular model of random graph exhibiting a cluster behavior. In the symmetric case, the graph has two equally sized clusters and vertices connect with probability p within clusters and q across clusters. In the past two decades, a large body of literature in statistics and computer science has focused on providing lower bounds on the scaling of |p - q| to ensure exact recovery. In this paper, we identify a sharp threshold phenomenon for exact recovery: if α = pn/log(n) and β = qn/log(n) are constant (with α> β), recovering the communities with high probability is possible if (α + β/2) - √αβ > 1 and is impossible if (α + β/2) - √αβ < 1. In particular, this improves the existing bounds. This also sets a new line of sight for efficient clustering algorithms. While maximum likelihood (ML) achieves the optimal threshold (by definition), it is in the worst case NP-hard. This paper proposes an efficient algorithm based on a semidefinite programming relaxation of ML, which is proved to succeed in recovering the communities close to the threshold, while numerical experiments suggest that it may achieve the threshold. An efficient algorithm that succeeds all the way down to the threshold is also obtained using a partial recovery algorithm combined with a local improvement procedure.

Original languageEnglish (US)
Article number7298436
Pages (from-to)471-487
Number of pages17
JournalIEEE Transactions on Information Theory
Volume62
Issue number1
DOIs
StatePublished - Jan 1 2016

Keywords

  • Clustering algorithms
  • Communities
  • Detection algorithms
  • Network theory (graphs)
  • Statistical learning

ASJC Scopus subject areas

  • Information Systems
  • Computer Science Applications
  • Library and Information Sciences

Fingerprint Dive into the research topics of 'Exact recovery in the stochastic block model'. Together they form a unique fingerprint.

  • Cite this