Algorithms for low-latency remote file synchronization

Hao Yan, Utku Irmak, Torsten Suei

    Research output: Chapter in Book/Report/Conference proceedingConference contribution


    The remote file synchronization problem is how to update an outdated version of afile located on one machine to the current version located on another machine with a minimal amount of network communication. It arises in many scenarios including web site mirroring, file system backup and replication, or web access over slow links. A widely used open-source tool called rsync uses a single round of messages to solve this problem (plus an initial round for exchanging meta information). While research has shown that significant additional savings in bandwidth are possible by using multiple rounds, such approaches are often not desirable due to network latencies, increased protocol complexity, and higher I/O and CPU overheads at the endpoints. In this paper, we study single-round synchronization techniques that achieve savings in bandwidth consumption while preserving many of the advantages of the rsync approach. In particular, we propose a new and simple algorithm for file synchronization based on set reconciliation techniques. We then show how to integrate sampling techniques into our approach in order to adoptively select the most suitable algorithm and parameter setting for a given data set. Experimental results on several data sets show that the resulting protocol gives significant benefits over rsync, particularly on data sets with high degrees of redundancy between the versions.

    Original languageEnglish (US)
    Title of host publicationINFOCOM 2008
    Subtitle of host publication27th IEEE Communications Society Conference on Computer Communications
    Number of pages9
    StatePublished - 2008
    EventINFOCOM 2008: 27th IEEE Communications Society Conference on Computer Communications - Phoenix, AZ, United States
    Duration: Apr 13 2008Apr 18 2008

    Publication series

    NameProceedings - IEEE INFOCOM
    ISSN (Print)0743-166X


    OtherINFOCOM 2008: 27th IEEE Communications Society Conference on Computer Communications
    Country/TerritoryUnited States
    CityPhoenix, AZ

    ASJC Scopus subject areas

    • General Computer Science
    • Electrical and Electronic Engineering


    Dive into the research topics of 'Algorithms for low-latency remote file synchronization'. Together they form a unique fingerprint.

    Cite this