Monitoring and Troubleshooting Faster Restores from Amazon S3 Glacier

This page describes tools for monitoring and troubleshooting faster restores for Amazon S3 Glacier.

Commvault Jobs

Each restore job has two jobs listed in the Commvault Command Center Job Manager:

• The initial restore job (parent)
• The Cloud Storage Archive Recall job (child)

Commvault Log Files

Active faster restores from S3 Glacier (which are initiated by the Cloud Storage Archive Recall workflow) cannot be monitored using the Job Monitor in Commvault Command Center. To monitor restore progress interactively, tail the following log files on the MediaAgent instance that is performing the restore:

• WorkflowEngine.log: On the instance that executes the Cloud Archive Recall workflow. Contains details of the initiation of the workflow.
• WorkflowCustom.log: On the instance that executes the Cloud Archive Recall workflow. Contains detailed parameters and the command-line options that are passed to the CloudChunkRecall.exe utility executing on the MediaAgent. Summarizes the number of successful and unsuccessful objects for troubleshooting partial or complete restore failures.
• CloudChunkRecall.log: Contains detailed information, including the Amazon S3 Batch Operations Job Id, the total number of objects requested, the total number of objects restored, and the total number of objects that remain.

Increasing to debug level = 3 allows detailed diagnostic reporting on the progress of the Cloud Archive Recall restore. - CloudActivity.log: Contains API requests made to the Amazon S3 service endpoints. Increasing to debug level = 3 will include full API request headers for troubleshooting. - CloudStats.log: Contains performance metrics on the volume of data transferred and the transfer speeds (throughput, latency) observed between the S3 endpoint and the Commvault MediaAgent.

Logging Amazon S3 API Calls Using AWS CloudTrail

Amazon S3 (including S3 Batch Operations) is integrated with AWS CloudTrail, a service that provides a record of actions taken by a user, a role, or an AWS service. You can use CloudTrail to observe the submission and execution of all API calls for the Amazon S3 restore, including the submission of the S3:CreateJob request.

You can view the most recent events in CloudTrail console in Event history.

You can also create an AWS CloudTrail for log retention and enable continuous delivery of CloudTrail events on Amazon S3 bucket.

Using the information collected by CloudTrail, you can determine the s3:CreateJob request that was made to Amazon S3, the IP address from which the request was made, who made the request, when the request was made, and additional details (such as the location of the Completion reports).

S3 Batch Operation Completion Report

Commvault requests Amazon S3 to write a completion report to the Commvault cloud storage location that the restore is performed from the following location:

s3://source-cloudstorage-bucket/s3BatchOperationsRestore/CVRestoreJobId-Commvault-Cloud-Storage-Archive-Recall-Workflow-Job-ID

Tracking Job Status and Completion Reports for Faster Restores from Amazon S3 Glacier

Commvault summarizes the number of successful and unsuccessful objects restored in Log Files/WorkflowCustom.log on the MediaAgent that performs the restore, for troubleshooting partial or complete restore failures.

To investigate failure root causes, you can use the Commvault Cloud Archive Recall Workflow Job Id (parent) to locate the appropriate CVRestoreJobId-nnn prefix and associated S3 Batch Operation completion report.

For more information about how S3 Batch Operation unsuccessful operations are handled and logged, see Tracking job failure.

The s3BatchOperationsRestore/CVRestoreJobId-Commvault-Cloud-Storage-Archive-Recall-Workflow-Job-ID folder contains a job folder for each s3:CreateJob that is executed. For example:

s3://source-bucket/s3BatchOperationsRestore/ 
   CVRestoreJobId-342/ 
      job-410b054c-be59-47ae-b04b-a713c148bedb/ 
         manifest.json 
         manifest.json.md5 
      results/ 
         e2ce4b092a4a670a58fa8d412e5a975658b5d49b.csv

For information about interpreting completion reports, see Examples: S3 Batch Operations completion reports.