As file systems grow larger, tools which were once industry standard become unsustainable at scale. Today, large data sets containing hundreds of millions of files often take longer to traverse than to copy. The time needed to replicate a file system has grown from hours to weeks, an unrealistic wait for a backup. Distsync is our new utility that can quickly update an out-of-date file system replica. By utilizing General Parallel File System (GPFS) policy scans, distsync finds changed files without navigating between directories. It can then parallelize work across multiple nodes, maximizing the performance of a GPFS. The National Energy Research Scientific Computing Center (NERSC) is currently using distsync to replicate file systems of over 100 million inodes and over four petabytes.
DistSync was presented as a Work in Progress (WIP) at the Parallel Data Storage Workshop, as well as in the SC’15 poster session. This work was done along with the storage systems group at the National Energy Research Scientific Computing Center (NERSC).
We developed distsync in order to facilitate the move from the Oakland Scientific Facility into a new building at the LBNL Berkeley campus. As part of this process, the scratch file system at the old location needed to be syncronized with an out of date replica present at the new location. Distsync dramatically reduced the time needed to process this syncronization.