Optimized Incremental Scans Using Qumulo Native Change File Tracking APIs

Updated

The snapshot capability of the Qumulo file system preserves the state of a file system directory at a point in time.

An incremental scan operation uses the REST APIs provided by Qumulo for snapshot creation and change file tracking to optimize the scan time. To determine which files changed, a Change File Tracking operation is performed between the previous backup snapshot and the current backup snapshot.

In addition, the Qumulo scan and backup operations read data off the snapshots to shift load from the production data shares. To offload production data shares, two snapshots are created and retained, one for the current backup operation and the other for the previous backup operation.

Data Set for CIFS Shares and NFS Exports

File count: 100 million

File size: Mixed file size

The following table compares performance metrics between regular scans and Qumulo file system scans of CIFS share and NFS export snapshots, of new files and modified files.

CIFS

Backup type

Items

Regular scan time

Qumulo CFT scan time

Full

Not applicable

02:31:00

02:39:00

Incremental

1 million, modified files

00:58:00

00:30:00

Incremental

1 million, new files under a new folder

01:20:00

00:03:40

NFS

Backup type

Items

Regular scan time

Qumulo CFT scan time

Full

Not applicable

05:04:00

03:07:00

Incremental

1 million, modified files

07:02:00

00:37:00

Incremental

1 million, new files under 1000 new folders

07:13:00

00:15:00