The job results directory size for Lustre (File System) depends on the CvScanDB size and the dirchange size.
All data access nodes in the Lustre cluster must have sufficient disk space for the job results directory.
CvScanDB Size Requirement Formula
Lustre Record Size
Fixed Record Size 197 bytes + Variables (File Name + Others) 259 bytes = 456 bytes
Average record size is 300 bytes and the index for each record is 100 bytes.
Each record average size = 300 bytes + 100 bytes = 400 bytes.
For 100 million files of one instance of CvScanDB, 37 GB space is required. There can be three instances of CvScanDB during the job duration.
So the recommended average size for CvScanDB for 100 million files in the Job Results directory is 37 x 3 = 111 GB.
Dirchange Size Requirement Formula
The following formula assumes that each folder is repeated only once in dirchange.
Dirchange Line Size
File Name 150 bytes + Tags 60 to 80 bytes = 230 bytes
For 100 million files, average dirchange space of 21 GB is required. There can be 3 dirchanges (full, incremental, and current) during the job duration.
So the recommended average size for dirchange for 100 million miles in the Job Result directory is 21 x 3 = 63 GB.
Total Recommended Size for Job Results Directory
The total recommended average size for 100 million files in the job results directory is 111 GB + 63 GB = approximately 174 to 200 GB.