Auxiliary Copy Troubleshooting

MM2000: Source MediaAgent cannot copy data to the destination MediaAgent with a different version

Auxiliary Copy operation might fail with the following error message:

 
Unable to get Source MediaAgent. Source MediaAgent cannot copy data to the destination MediaAgent with a different version.
 Source: CommServe01, Process JobManager

Symptom

This error message appears and the Auxiliary Copy operation might fail if the source and destination MediaAgents are with different release version.

Resolution

Make sure that the source and destination MediaAgents are in same software release version.

MM2001: Auxiliary Copy Job Goes To Waiting State

Symptom

The Auxiliary Copy job goes into Waiting state due to one of the following reasons:

No Resources Available

The Media is not available

Resolution

The Auxiliary Copy job will automatically resume back in few minutes and continue once the resources becomes available.

MM2002: Auxiliary Copy Job Failure

Symptom

Auxiliary Copy job failed with the following error message:

Loss of Control Process for AuxcopyMgr.exe

Resolution

If you have scheduled large number of Auxiliary Copy jobs (e.g., more than 100 jobs) to run on CommServe and if the job fails with loss of control error message due to system memory limit, then perform the following to solve the issue:

1. Apply latest service pack on CommServe computer.

2. On the CommServe computer, modify the Windows Desktop Memory settings by editing the following registry key:

   1. Click Start, click Run, type regedit, and then click OK.

   2. Navigate to the following directory:

      HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems

   3. Right click the Windows and click Modify.

   4. In the Value data, modify the value 768 in SharedSection to 2048 (2 MB)

      For example, if the value of SharedSection is defined as 1024, 20480, 768 modify 768 value to 2048. After modification the value will appear as 1024, 20480, 2048.

   5. Click OK.

3. Reboot the CommServe computer.

For more information on memory issues, refer to Microsoft article at http://support.microsoft.com/kb/126962

MM2004: Auxiliary Copy Job goes to Pending State

Symptom 1

Auxiliary copy job goes to a Pending state and the Reasons for Job Delay box on the General tab of the Admin Job Detail dialog box displays one of the following reasons:

Source Media Error/ Source Media not Available

Read Error

Target Media Mount Error

Symptom 2

Auxiliary copy goes to a Pending state if resource allocation does not go through within 1 hour. The Reasons for Job Delay box on the General tab of the Admin Job Detail dialog box displays the following reason:

Error Code: [19:2054] Description: Unable to allocate resources. Please check the Progress tab for more details.

Resolution

Perform the following:

1. On the ribbon in the CommCell Console, click the Storage tab, and then click Media Management.

2. Click the Auxiliary Copy Configuration tab.

3. In the Skip job on read errors during Auxiliary Copy box, type 1 to skip jobs during Auxiliary copy operation.

4. Click OK.

The following table explains how the auxiliary copy job will be performed when an error occurs:

MM2005: Completed w/ one or more errors

Symptom

Auxiliary Copy operations are completed in the Job History with the error message.

Resolution

This issue occurs due to one of the following:

• When Auxiliary Copy operation encounters jobs with Status as Bad on the source storage policy copy. In this scenario, the jobs that are marked as Bad and any jobs referring to these jobs will be skipped and disabled for Copy on the destination Storage Policy copy.

MM2006: Failed to copy chunk or verify chunk

The Auxiliary copy operation fails with one of the following error message:

Error Code: 40:91
 Description: Failed to Copy or verify Chunk [XXX] in media [XXX], Storage Policy [XXX], Copy [XXX], Host [XXX], Path [XXX\XXX], File Number [xxx], Backup Jobs [xxxxx ]. Data read from media appears to be corrupted.

Failed to copy or verify chunk [...] in media [...] for storage policy [...] copy [...]: Backup job [...].
 Data read from media appears to be corrupted.

Symptom

The error message appears and the Auxiliary Copy operation fails if the source data is invalid. This might occur if the source disk library has been damaged.

Resolution

If the source data is invalid, then the job being copied must be disabled for copy in order for the Auxiliary Copy to complete.

1. To determine which job to disable, examine the error in the Event Viewer.

   The Job ID that is listed in the Backup Jobs part of the error message is the job that needs to be disabled.

   Failed to Copy or verify Chunk [...] in media [...], Storage Policy [...], Copy [...], Host [...], Path [...], File Number [...], Backup Jobs[...].

2. Disable the job from the Storage Policy Copy.

   For more information, see Prevent Jobs from Being Copied.

Symptom

The error message appears and the tape-to-tape Auxiliary Copy operation fails if the tape has been bad.

Resolution

If the Source tape is bad, find the jobs (Job ID) that were written to the bad tape from the error message in the Event Viewer.

Error Code: 40:91
 Description: Failed to Copy or verify Chunk [XXX] in media [XXX], Storage Policy [XXX], Copy [XXX], Host [XXX], Path [XXX\XXX], File Number [xxx], Backup Jobs [xxxxx ]. Data read from media appears to be corrupted.

1. From the CommCell Browser, select Storage Resources | Libraries | <Library Name> | Media By Groups.

   Find the tape in the Expired Media or Assigned Media.

2. Once you locate the appropriate media, right-click the media, point to View and then click View Content.

   The Data on Media dialog box appears.

3. Find the date of the latest job and note down the date of the job and then click Close.

4. In the Job Controller window, right-click the Auxiliary Copy job and then click Kill.

5. From the CommCell Browser, select Policies | Storage Policies | <Storage Policy Name>.

6. Right-click the appropriate storage policy copy, and then click Properties.

   The Properties dialog box appears.

7. Click the Copy Policy tab.

8. Clear the All Backups check box, and select the Backups On and After date that is the date of the latest backup job.

9. Click OK.

   A message appears that asks you if you want to change the date.

10. Click Yes.

11. Restart the Auxiliary Copy job.

    Right-clicking the Storage Policy, point to All Tasks and then click Run Auxiliary Copy.

MM2007: Operating System could not find the specified path

Symptom

The Auxiliary Copy operation fails with the following error message:

Error Code 40:91:
 Failed to copy or verify chunk [...] in media [...] for storage policy [...] copy [...]: Backup job [...]. Operating System could not find the path specified.

Error Code 62:411
 Operating System could not find the path specified.

Cause

This error message may appear if the source data chunk is not present at the indicated path due to one of the following reasons.

1. Data was accidentally deleted or Disk volume or folder was moved.

2. Disk device no longer visible by Operating System or drive letter has changed.

3. SAN drives may have gotten swapped following a reboot.

Resolution

1. Determine whether the source disk volume is available at its original location and drive.

2. Verify that the drive still has the CV_MAGNETIC folder at the proper location.

3. If the CV_MAGNETIC folder exists, open the folder and verify that the V_# and CHUNK_# folder and paths are still available.

   If the folders are not present at the location, it may have been deleted or written to a different volume while the volumes were swapped.

4. On SAN volumes, when the system is restarted, the drives get assigned wrong letters by the Operating System.

   If the drives get assigned wrong letters, then you need to determine if the current configuration is correct and the problem occurred in the path.

   1. From the CommCell Browser, expand Storage Resources | Libraries.

   2. Right-click library and then click Properties.

   3. On the Sharing tab, select the appropriate device and then click Edit.

   4. In the Folder box, verify the location and path structure.

      If the path is incorrect, use Browse to locate the path and reselect.

      For network location, verify username and password by re-entering the credentials.

      Example of incorrect path: Path displays \\\\ instead of \\.

   5. Click OK.

5. If the current configuration is correct and the backups were written to the wrong volume, perform a search for the V_# folder on the system and move them back to the correct volume to allow the Auxiliary Copy to progress.

   Alternatively, if the current configuration is incorrect, then swap the volumes back and then allow the Auxiliary Copy operation to resume.

MM2008: Auxiliary Copy fails when using Data Domain device

The Auxiliary Copy operation fails with the following error message:

Error Code 13:20:
 Failed to read chunk [...] in media [...] for storage policy [...] copy [...]: Backup job [...]. Encountered an I/O error while performing the operation

Error Code 62:342:
 Encountered an I/O error while performing the operation.

Symptom

The above error message appears and the Auxiliary Copy operation fails when it is performed on a storage policy in which the Primary Copy is stored on a Data Domain device.

Possible Causes

• If the Data Domain is configured as Workgroup.

• If the account used to access the mount paths on the Data Domain device does not have rights assigned to the following default configuration path on the MediaAgents then the job(s) might fail.

  C:\Program Files\CommVault\Commvault

Resolution

1. Add the user/service account which was used to configure the UNC/CIFS mount path for the Data Domain to the local administrators group of the source and destination MediaAgents.

2. Make sure the user/service account has Full Rights to the installation directory. For Windows Vista, Windows 7, Windows 2008 and Windows 2008 R2 requires explicit rights are assigned to this folder, including all subfolders and files.

   This is not usually required if the Data Domain is configured to authenticate user accounts using Active Directory.

   To see what user/service account is configured on the mount path, see Modify the Mount Path.

   Note: In the Connect As box, the user account is the account that needs to be added in the local administrators group

MM2009: Failed to retrieve drive pool information

The Auxiliary Copy operation fails with the following error message.

Error Code 62:264:
 Failed to retrieve drive pool information.

 
Error Code: 32:254:
 Description: Read operation from media [...] could not be completed. The reason: Read operation cannot be completed as none of the resources available can be used. Please make check the following: 1. There are drives with compatible recording format to the media being read. 2. There are resources (Drives, Libraries, MediaAgents, and so on.) that are online and ready to be used. If you still have an issue, please contact customer support.

Symptom

The above error message appears and the Auxiliary Copy operation might fail, if the data path of the storage policy copy is not defined properly.

Cause

To verify the data path:

1. From the CommCell Console, select Policies | Storage Policies | <Storage Policy Name>

2. Select and right-click the appropriate storage policy copy, and then click Properties.

3. Select the Data Path tab.

4. Verify the status of the Data Path(s) available to use and that the MediaAgent has access to the required Data Path(s).

5. Click OK.

MM2010: Skipped Jobs during the Auxiliary Copy Operation

Symptom

During Auxiliary copy operation, a few jobs might not get copied to the secondary copy and the following event message is displayed in the Event Viewer

Some archive files from jobs [...] have been skipped on copy [SP/copyName] since old chunks from these archive files have not been fully pruned

Cause

The job listed in the event message might not get copied because these jobs may not have been aged from the destination DDB when copy operation was performed.

Resolution

The jobs will be copied to the secondary copy when they are completely aged from the CommServe database.

MM2011: Recopy already copied jobs

Symptom

The Auxiliary Copy job displays the following error message:

Auxiliary Copy Policy destination media is bad

Cause

An Auxiliary Copy operation encounters bad media but has already copied some data to the media before the job started receiving failed to copy data errors.

When the destination media is bad the data copied to the bad tape will not be re-copied to the new tape.

Resolution

To re-copy those jobs associated with a failed Auxiliary Copy job where the destination media is bad follow the steps below.

1. From the CommCell Browser, select Policies | Storage Policies <Storage Policy Name>

2. Right-click the storage policy copy, point to View, and then click Jobs.

   The jobs that already copied appears as Available.

3. Select and right-click the jobs that are available that were copied to the bad tape, and then click Re-Copy.

4. A message appears, that asks you that do you want to copy the jobs. Click Yes.

Symptom

Re-Copy partially copied jobs

Resolution

To Re-Copy partially copied jobs:

1. From the CommCell Browser, select Policies | Storage Policies <Storage Policy Name>

2. Right-click the storage policy copy, point to View, and then click Jobs.

   The jobs that are partially copied is displayed as Partially Copied.

3. Select and right-click the jobs that are partially copied to the media, and then click Re-Copy.

4. A message appears, that asks you that do you want to copy the jobs. Click Yes.

   Note

   Data can only be copied if it still exists in the source copy.

MM2012: Auxiliary Copy job uses too many tapes

Symptom

When performing an Auxiliary Copy operation the job uses too many tapes.

Cause

The Auxiliary Copy uses too many tapes because the Storage Policy is set to use multiple streams and the primary copy is configured with disk.

When the Auxiliary Copy runs it will try and reserve the same amount of media based on the amount of streams configured in the Storage Policy. You many notice that many tapes were used during the Auxiliary Copy with only a small amount of data written to each tape.

Resolution

To resolve this issue:

1. From the CommCell Browser, select Policies | Storage Policies | <Storage Policy Name>.

2. Right-click the appropriate secondary copy, and then click Properties.

3. Click the Media tab.

4. Select Combine source data streams to check box, and then enter value in the Stream(s) box.

   When this option is enabled the Auxiliary Copy operation will only use the amount of drives (streams) specified in the Stream(s) box at one time.

   That is, if it is configured to Combine source data streams to 1 streams it will use one tape at a time until it fills that tape and then moves on the next media.

5. Click OK.

6. A message appears, that tells you that combine streams for destination copy with tape will cause restore issues for backups of SQL and Sybase agent subclients. This is because the data recovery operation for SQL and Sybase agents is not supported from the combined streams on tape.

   The data from this destination copy must be copied to a disk copy, and then restores should be performed from that disk copy.

7. Click OK to continue.

   Note

   Do not exceed the number of physical drives in your tape library.

MM2013: Restore job fails when media is reserved for other jobs

Symptom

A restore job fails with the following error message:

Error Code 62:348:
 The media is already Reserved by some other Job[s]

Cause

This error message appears for database restore when the primary copy backup is multi-stream and during the Auxiliary Copy operation the Storage Policy copy is set to Combine streams.

To check the configuration for the Auxiliary copy operation:

1. From the CommCell Browser, select Policies | Storage Policies | <Storage Policy Name>.

2. Right-click the Secondary Copy, and then click Properties.

3. Click the Media tab.

   Note down the number of streams entered in the Combine source data streams to [ ] Stream(s) box.

Resolution

To resolve this issue, you must recover the data from the secondary copy (secondary copy on tape) to secondary copy on the disk, and then perform restores from this secondary copy (configured on disk).

Phase 1: Recover job from Auxiliary Copy on tape library to Auxiliary Copy on disk library

Phase 2: Restore of disk library Auxiliary Copy backup database data to the destination server

• View the job to be restored:

  1. From the CommCell Browser, select Policies | Storage Policies | <Storage Policy Name>.

  2. Right-click the Secondary Copy that contains the data to be restored, point to View, and then click Jobs.

  3. Clear Include All Client Computer check box.

  4. Exclude all clients except the client to be restored.

  5. Clear Specify Time Range, and then click OK.

  6. In the Jobs for Storage Policy Copy window, find the specific job and time range to restore from the job ID and Start Time.

  7. Right-click the job, and then click View Media to assure the correct tapes.

• Create a new storage policy copy with the disk library

  1. Right-click the Storage Policy, point to All Tasks, and then click Create New Copy.

  2. In the Copy Name box, type the name for a copy.

  3. From the Library and the MediaAgent list, select disk library and the MediaAgent hosting the disk library.

  4. Click the Copy Policy tab.

  5. Clear All Backups check box.

  6. In the Backups On and After, select the date before the date of the job to restore.

     For example, a full backup to be restore id 08/18/2013, in the Backups On and After set to date 07/01/2013.

  7. Select Specify Source for Auxiliary Copy check box, and then select the restore job's original secondary copy.

  8. Click the Association tab.

  9. Clear all the clients, and then select only the client being restored.

  10. Click OK to create the policy.

  11. Right-click the newly created secondary copy, point to View and then click Jobs.

  12. Clear Specify Time Range, and then click OK.

  13. The jobs that need to be copied appears in the Jobs for Storage Policy Copy window.

  14. Select all the job that are not to be restored.

  15. Right-click the selected jobs and then click Do Not Copy to prevent these jobs from being copied to the disk library

• Run the Auxiliary Copy

  1. Right-click the Storage Policy, point to All Tasks, and then click Run Auxiliary Copy.

  2. Click Select A Copy, and then select the newly created copy.

  3. An Auxiliary Copy job appears in the Job Controller window which initiates the copy of Job from Tape to Disk.

  4. Once the Auxiliary Copy job is complete, right-click the copy, point to View, and then click Jobs.

  5. Clear Specify Time Range, and then click OK.

  6. Select and right-click the job to be restored, click Browse and Restore to initiate the restore the files required from the job list.

     For more information, refer restore procedure in the agent section.

MM2014: Auxiliary Copy Re-Initialization

Symptom

The following event message appears in the Event Viewer during Auxiliary Copy operation:

Some backup jobs completed or some backup jobs were selected after this job started. Start copying/verifying more data.

Resolution

This message appears if you have configured the Auxiliary Copy operation to copy first full backup jobs of the subclient, and then copy the rest of the backup jobs.

See Configure Auxiliary Copy to Copy First Full Backups of the Subclient for more information.

This message tells you that Auxiliary Copy job has copied all available backup jobs to a secondary storage. Before marking this Auxiliary Copy job to complete, it verifies if there are any new backup jobs available when Auxiliary Copy job was initiated. If so, this Auxiliary Copy job generates the above event message and copies those new backup jobs to a secondary storage.

MM2015: Subclient Association to a Storage Policy Fails

Symptom

The following error message appears when user tries to associate a subclient to a storage policy and if the data to be copied is more than three times the specified value in the Auxiliary Copy Fallen Behind criteria, or if any of the other specified criteria is exceeded by more than three times.

Operation successful. However, the storage policy has significant amount of pending data to copy. Please review documentation on "Resolving the Auxiliary Copy Fallen Behind Issues" for more info.

Resolution

Perform the auxiliary copy operation to copy the data that is pending a copy operation.

For instructions about resolving Auxiliary Copy fallen behind alert issues, see Resolving the Auxiliary Copy Fallen Behind Alert Issues.

MM2016: Error Code [32:408] There is no more data to process currently. Rest of the unprocessed data are either being sent to MediaAgent or skipped due to failures

Symptom

The auxiliary copy operation hangs in a pending state with the following error message:

 
Error Code [32:408] There is no more data to process currently. Rest of the unprocessed data are either being sent to MediaAgent or skipped due to failures.

Cause

This is a general error message that is displayed when enough resources are not available to run the auxiliary copy operations. There are many reasons behind unavailability of resources. For example, the MediaAgent is offline.

Resolution

The detailed reason for the auxiliary copy job hanging in pending state is displayed on the Progress tab of the job window.

1. From the CommCell Console ribbon, on the Home tab, click Job Controller.

2. Double-click the auxiliary copy job and click the Progress tab.

AUXCP0010: Unsupported Data Path selected for Copy

For more information, see KB Article AUXCP0010.

AUXCP0011: Auxiliary copy operation completes with "Completed W/ One or More Warnings" message

For more information, see KB Article AUXCP0011.

AUXCP0014: Error Code: [40:61] Auxiliary Copy Cannot Write Multiplexed Data on to a Deduplication Enabled Library as this Operation is Not Supported

For more information, see KB Article AUXCP0014.