When Commvault backs up a Hive instance, it handles managed and external tables differently. External tables are read directly from HDFS, but managed tables must first be exported to an HDFS staging location before they can be backed up.
By default, Commvault exports all managed tables to the staging location before starting the backup. This behavior requires enough staging space to hold every managed table at the same time.
If your environment doesn't have enough staging space, you can enable optimized staging. Optimized staging exports and backs up managed tables in batches, reducing the amount of HDFS staging space that the backup requires.
How optimized staging works
Optimized staging exports managed tables according to the available staging space.
The number of tables that Commvault exports simultaneously depends on the available staging space.
-
Commvault always exports the largest table first.
-
After exporting the largest table, Commvault checks whether another table fits in the remaining staging space.
-
If another table fits, Commvault exports and backs up the tables in parallel.
-
After a table is backed up, Commvault removes the staged data and exports the next table.
Tip
Create a server group that contains all Hive nodes, and apply these settings to the server group.
Configure optimized staging
Enable optimized staging by applying the following additional settings.
| Setting | Purpose |
|---|---|
| nOptimizeHiveExport | Enables optimized staging by using 10% of the available free space in the HDFS staging location. |
| nHiveStagingSpaceInGB | Overrides the 10% calculation and sets a fixed staging space size in GB. |
| nHiveMaxExports | Limits the maximum number of tables that Commvault exports simultaneously. |
Enable nOptimizeHiveExport to use optimized staging. Optionally configure nHiveStagingSpaceInGB or nHiveMaxExports to further control staging behavior.
nOptimizeHiveExport
nOptimizeHiveExport enables optimized staging.
When you apply this setting, Commvault uses 10% of the available free space in the HDFS staging location as the staging space. It exports the largest table first, backs it up, removes it from the staging location, and repeats the process for the remaining tables. If enough staging space is available, Commvault exports tables in batches.
nHiveStagingSpaceInGB
nHiveStagingSpaceInGB sets a fixed staging space size.
By default, optimized staging uses 10% of the available free space in the HDFS staging location. When you also configure nHiveStagingSpaceInGB, Commvault ignores the 10% calculation and limits the staging space to the value that you specify, in GB.
Set this value higher than the size of your largest managed table.
If the configured staging space cannot hold a table, Commvault exports and backs up every table that fits within the configured staging space. It then attempts to export the larger table. If the export command fails, Commvault skips that table and marks the backup job as partially complete.
nHiveMaxExports
nHiveMaxExports enables parallel exports.
By default, optimized staging determines how many tables to export based on the available staging space. When you also configure nHiveMaxExports, you specify the maximum number of tables that Commvault exports simultaneously.
The actual number of tables exported simultaneously depends on both the available staging space and the value configured for nHiveMaxExports.
Example
The following example shows how optimized staging uses 30 GB of staging space to back up five managed tables.
| Table | Size |
|---|---|
| Table 1 | 8 GB |
| Table 2 | 15 GB |
| Table 3 | 7 GB |
| Table 4 | 11 GB |
| Table 5 | 9 GB |
Commvault first exports the 15 GB, 8 GB, and 7 GB tables because they exactly fill the available staging space.
After the 15 GB table finishes, Commvault frees that staging space and exports the 11 GB table.
If the 8 GB table finishes first instead, only 8 GB becomes available. Because neither the 11 GB table nor the 9 GB table fits in the remaining space, Commvault waits until enough staging space becomes available before exporting another table.