Rapid Replication of Multi-Petabyte File Systems

DistSync was presented as a Work in Progress (WIP) at the Parallel Data Storage Workshop, as well as in the SC’15 poster session. This work was done along with the storage systems group at the National Energy Research Scientific Computing Center (NERSC).

We developed distsync in order to facilitate the move from the Oakland Scientific Facility into a new building at the LBNL Berkeley campus. As part of this process, the scratch file system at the old location needed to be syncronized with an out of date replica present at the new location. Distsync dramatically reduced the time needed to process this syncronization.