Backup Troubleshooting - Windows File System iDataAgent

Table of Contents
  • WFS0001: Backup failure while backing up large files or system volumes
  • WFS0002: Backup goes to pending state while backing up an unformatted volume and mount points
  • WFS0003: Backup goes to pending state for Polyserve (Non-Microsoft clusters)
  • WFS0004: Backup of subclient that contains only files and no folders
  • WFS0005: Completed with one or more Errors
  • WFS0006: Full backup of Lync server when Incremental or Differential backup is performed
  • WFS0007: Lync server data and settings are not backed up
  • WFS0008: File System scan failed - there is nothing to backup
  • WFS0009: Failure of File System scan
  • WFS0010: Synthetic Full backup failed
  • WFS0011: Files appear corrupted
  • WFS0012: Potential sparse file corruption on Windows operating system
  • WFS0013: Incremental Backups Are of the Same Size As That of a Full Backup
  • WFS0014: Data protection error: [Err:1450]"Insufficient system resources exist to complete the requested service"
  • WFS0015: Backup failures on Windows 2008, Windows 2008 R2 or Windows 7
  • WFS0016: Backups may fail to back up Windows System State "Event Logs"
  • WFS0017: Backup job pending with error: "File Scan failed to write to dirchange file"
  • WFS0018: System state backup components fail
  • WFS0019: VSS errors
  • WFS0032: Skip deleted or missing files during backup
  • WFS0033: Contents are not backed up although the backup completes successfully
  • UFS:0002: Error registering job with the Communications Service
  • UFS:0005 After an authentication failed error if a MediaAgent or a client cannot communicate with CommServe it will disable itself immediately
  • WFS0034 Windows operating system failures
  • WFS0035: Backup failure of cluster computers
  • WFS0036: Large Size Incremental Backups
  • WFS0037: Backup Using Post-Backup Process Fails because of Using a Different MediaAgent

WFS0001: Backup failure while backing up large files or system volumes

Cause

When backing up large files and/or system volumes, the Windows memory pool resources may run low and the backup may fail with errors reported in the Windows Event Viewer and the iDataAgent logs.

Solution

Review the Microsoft documentation for increasing the maximum memory pool usage and paged memory pool size.

WFS0002: Backup goes to pending state while backing up an unformatted volume and mount points

Symptom

The Event Viewer displays following message: Cannot scan <path>

Solution

Add the NSkipDirErrors Registry Key and set the value at 1 (or any value other than 0)

WFS0003: Backup goes to pending state for Polyserve (Non-Microsoft clusters)

Symptom

The backup job for a Polyserve goes to pending state

Solution

Exclude the following folders from the backup and anti-virus scan:

For backup c:\program files\polyserve\matrixserver\conf\mounts
For anti virus scans c:\program files\polyserve\matrixserver\conf

<Simpana install directory>/Base

During Disaster Recovery, open up the Polyserve Configuration Utility and export the configuration to the node that was rebuilt. This will ensure complete synchronization prior to introducing a rebuilt node again into the matrix.

WFS0004: Backup of subclient that contains only files and no folders

Cause

When a subclient contains only files and no folders, the backup can fail if you select Optimized Scan as the scan method.

Solution

When a subclient contains only files and no folders, always use Recursive Scan as the scan method.

WFS0005: Completed with one or more errors

Backup jobs from Windows File System iDataAgent will be displayed as "Completed w/ one or more errors" in the Job History in the following cases:

For subclient containing only system state backup:

  • Fails to backup critical component - the job status is failed
  • Fails to backup non-critical component - the job status is completed w/ one or more errors

For subclient containing system state backup along with File system data:

  • Failure to backup critical component - marks the job as completed w/ one or more errors, so the data can continue to backup.
  • Failure to backup non-critical component - marks the job as completed w/ one or more errors

WFS0006: Full backup of Lync server when Incremental or Differential backup is performed

When a subclient is configured for a Lync server, always a full backup of the data and settings is performed, irrespective of the backup type selected for the subclient.

WFS0007: Lync server data and settings are not backed up

The Lync server must be running when you perform backup of the subclient configured for the Lync server.

WFS0008: File System scan failed - There is nothing to backup

Cause

Synthetic full backup will fail with the error message File System scan failed - There is nothing to backup if the Check for deleted stubs option was enabled in the Advanced Backup options dialog box.

Solution

Make sure that the Check for deleted stubs option is not selected while initiating a synthetic full backup. To disable this option use the following steps:

  1. From the CommCell Browser, navigate to Client Computers | <Client> | File System | defaultBackupSet.
  2. Right click the subclient and point to Backup.
  3. In the Backup Options for Subclient dialog box, select synthetic full option.
  4. Click Advanced and make sure that the Check for deleted stubs option is not selected in the Advanced Backup options dialog box..

WFS0009: Failure of File System scan

Cause

Backup operation will fail at the scan phase if you have OPSWAT Secure RAM disk software installed on your computer that is being backed up. This is because the virtual disk created by the OPSWAT Secure RAM disk software is not being identified as a RAM disk thereby causing the file system scan to fail.

Solution 1

The virtual disk created by the OPSWAT Secure RAM disk software must be excluded from the subclient content by adding the virtual drive to the list of filters. For more information on adding content to filters, see Configuring Filters for Backups.

Solution 2

Specify the drives that need to be backed up manually in the subclient content.

WFS0010: Synthetic Full backup failed

If an absolute path is given as the subclient content and the path does not contain any files/folders, then the full backup would complete successfully but the subsequent synthetic full backup would fail.

WFS0011: Files appear corrupted

Backup files that are created by using the Windows Volume Shadow Copy feature are corrupted in Windows Server 2003, Windows Vista, Windows Server 2008, Windows 7 and Windows Server 2008 R2.

This is a Microsoft known issue, see Microsoft KB 2748349 for more information.

WFS0012: Potential sparse file corruption on Windows operating system

Symptom

Data corruption occurs on a computer operating on Windows 7 or Windows 2008 R2 when a sparse file undergoes random write stress, and the data in the sparse file is overwritten with 64 KB of zeros. This happens due to a rare condition in the NTFS file system driver. This behavior causes data corruption under heavy write stress for sparse files.

You may receive the following error codes:

Error Code 9:30

[...]:Remote system [...]. The socket is in an invalid state [...] for desired action

Error Code 13:127

Error occurred while processing chunk [...] in media [...] for storage policy [...] copy [...]: Backup job [ ... ]. Data read from media appears to be corrupted.

Error Code 62:469

Data read from media appears to be corrupted

The Tag Header for all logs will always return a NULL or =Blankspace value as follows:

  • For a restore or a synthetic full job, the following errors would appear in CVD.log on the MediaAgent

    8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] **ERROR** DataMoverSingleInstancer::ReadTagHeader: Failed to convert the tag header to Host byte order. Tag header got =
    8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] The current Read offset into the chunk Physical 11602 Logical 0
    8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] Metadata file [F:\Backup\CV_MAGNETIC\V_1211796\CHUNK_5018060\CHUNK_META_DATA_5018060].
    8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] Data file [F:\Backup\CV_MAGNETIC\V_1211796\CHUNK_5018060\SFILE_CONTAINER_001]. Version [2], SFile Number [1], Last Seek Offset [904877]
    8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] Current data file offset [904973]

  • For an Auxiliary Copy job, the following errors would appear in AuxCopy.log on the MediaAgent

    8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] **ERROR** DataMoverSingleInstancer::ReadTagHeader: Failed to convert the tag header to Host byte order. Tag header got =
    8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] The current Read offset into the chunk Physical 11602 Logical 0
    8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] Metadata file [F:\Backup\CV_MAGNETIC\V_1211796\CHUNK_5018060\CHUNK_META_DATA_5018060].
    8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] Data file [F:\Backup\CV_MAGNETIC\V_1211796\CHUNK_5018060\SFILE_CONTAINER_001]. Version [2], SFile Number [1], Last Seek Offset [904877]
    8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] Current data file offset [904973]

Additional logs available on the MediaAgent

  • CVD.log

    1896 164 07/25 08:53:02 1007062 #--1 [DM_BASE ] Opening the Chunk =6322105, ArchFileId = 1008183, FileMarker=623, ArchFilePhysSizeInChunk=2146389541 VolId=744525
    1896 164 07/25 08:53:02 1007062 #--1 [MEDIAFS ] Magnetic Mount Path :- KBRemaning 560559808, Minimum space MB to be left [2048], maximum space MB to be filled [-1]
    1896 164 07/25 08:53:02 1007062 #--1 [MEDIAFS ] RealMagneticFS Opened file
    1896 160c 07/25 08:53:16 1007062 [UNCOMPRESS ] #UNCOMPRESSION ERROR! rc = -3 BN = 0 Scheme = 0 InputSize: 62974 UncompressedSize: 63392 ExpectedSize: 63392
    1896 164 07/25 08:53:26 ####### [CVD ] STOP_PL Control Message Received 0 (Pipeline Thread self-terminated)
    1896 1408 07/25 08:53:26 1007062 [DECRYPT ] STOP_PL while allocating a buffer
    1896 164 07/25 08:53:26 1007062 [FSRESTHEAD ] [FsRestoreHead::stopPipeline]: Received STOP_PL control message. Bailing out…

    1896 1df8 08/10 08:33:35 #######[ NULL] invalid tag header 1896 1df8 08/10 08:33:35 1016869 #--1 [DM_BASE ] ERROR DataMoverBase::Read: Failed to convert the tag header to Host byte order. Tag header got = 1896 1df8 08/10 08:33:35 1016869 #--1 [DM_BASE ] The current Read offset into the chunk Physical 1077840646652 Logical 1183781091958 1896 1df8 08/10 08:33:35 1016869 [DM_READER ] DataReader::Read: Failed to READ data from ARCHIVE FILE [1010462] COPY [134]. PHYSICAL LEN = [96] LOGICAL LEN =[0]. TAGHEADER = [1]
    1896 1df8 08/10 08:33:35 1016869 [FSRESTHEAD ] Encountered error when reading data from the media.
     

  • FSIndexedRestore.log

    3164 1250 07/25 08:53:32 1007062 CVRestoreFiles::restoreFile() - Sending FSR_MSG_OPEN_ARCHIVE (2,18,1009296)
    3164 1250 07/25 08:53:32 1007062 CVRestoreFiles::sendMsgToClient() - CVSession::sendMessage() returned error [900001e=[CVSession::sendMessage]: Remote system [...]. The socket is in an invalid state [CVS_STATE_INIT] for desired action.]
    3164 1250 07/25 08:53:33 1007062 RSTCALLBACK: error restoring '\D:\Users\...\My Documents\DOCUMENTS\documents for publication\filename' afile id (2,18,1009296) offset hi 0 lo 1173355286
    3164 1250 07/25 08:53:33 1007062 RSTCALLBACK: cannot restore node 1571547 version 0 afile 352 offset 1173355286
    3164 1250 07/25 08:53:33 1007062 0 RESTORE: ERROR: Restore Failed for ...\Windows File System\defaultBackupSet\D Users(CommCell:AppNum).
    3164 1250 07/25 08:53:33 1007062 0 RESTORE: Source path [\] ([1] of [1]).

Solution

We recommend you to obtain the Microsoft hot fix for this issue. Refer to Microsoft KB 270881 to obtain the hotfix.

WINFS:0013 Incremental Backups Are of the Same Size As That of a Full Backup

Symptom

If you have installed Symantec Antivirus software on your client computer, when you perform an incremental backup followed by a full backup, the size of the incremental backup is the same as that of the full backup. Also, you may notice that a large amount of data is backed up.

Cause

This issue is due to the Symantec Antivirus software installed on your client computer. After Symantec Endpoint Protection performs a manual or scheduled scan, the access or modified time of the file is changed. As a result, the files are backed up again.

Solution

Create the NoFileMod setting as follows:

  • On a 32-bit version of a client computer that has Symantec Endpoint Protection installed on it, create the following DWORD value and set it to 1. HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Symantec Endpoint Protection\AV\NoFileMod
  • On a 64-bit version of a client computer that has Symantec Endpoint Protection installed on it, create the following DWORD value and set it to 1. HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Symantec\Symantec Endpoint Protection\AV\NoFileMod

For more information, see Symantec Endpoint Protection scan causes incremental backups to perform full backups.

WINFS0014: Data protection error: [Err:1450]"Insufficient system resources exist to complete the requested service"

Symptom

Data protection jobs may run slowly, fail, or fail to backup some large file with the following errors.

  • From clbackup.log on the client server.

Error [1450] during BackupRead. Err:[1450] Insufficient system resources exist to complete the requested service.

ERROR 1130: Not enough server storage is available to process this command.

  • Operating system Application Event Logs (When you use a third party backup software that uses NT Backup API)

ERROR 1450: Insufficient system resources exist to complete the requested service.

ERROR 1450: / hex 0x5aa ERROR_NO_SYSTEM_RESOURCES

Operating system error 1450 Insufficient system resources exist to complete the requested service.

Write on "device" failed, status = 1450

ERROR 1130: Not enough server storage is available to process this command.

ERROR 1130 / hex 0x46a ERROR_NOT_ENOUGH_SERVER_MEMORY

Backup or restore operation terminating abnormally.

Solution

This is a Microsoft known issue. Refer to Microsoft KB article 304101 for more information.

Additionally, you can also look at Microsoft KB articles 2002606 and 967351 for more details.

WINFS0015: Backup failures on Windows 2008, Windows 2008 R2 or Windows 7

Symptom

The following errors may be generated during the backup operation:

  • From CommCell Console Event Viewer or Job Details dialog box:

Error Code 17:55:
Failed to create a VSS snapshot and it is required in order to run a system state backup. Failing the scan.
FileScan Error:Cannot scan [\]
 

  • From the FileScan.log file on the Client server:

CsVolumeList::Add() - VolGUID path (\\?\Volume{0c022041-ff03-11de-8b2f-806e6f6e6963}\Boot) too long. Truncating.
GetVolumeMountPointsForVolumeGUID() - Found no path matching vol name [\\?\Volume{0c022041-ff03-11de-8b2f-806e6f6e6963}\].
CsVolumeList::Add() - Not able to get volume mount points for vol [\\?\Volume{0c022041-ff03-11de-8b2f-806e6f6e6963}].
CsVolumeList::Add() - Added [\\?\VOLUME{0C022041-FF03-11DE-8B2F-806E6F6E6963}\] to volume list

Once the FileScan.log shows the above error, then system state VSS snap is started for the file scan and the following error message is seen when the files scan or backup fails.

The cause for the failure is that VSS is unable to create a SNAPSHOT of the volume identified.

You may observe the following in the logs:

CsSnapRequestor::StartSnapshotSet() - Starting a new Snapshot Set
Entering [CsVssBackupComponents::StartShadowSet]
CsSnapRequestor::StartSnapshotSet() - Created shadow set e4875fdb-5caa-48ed-abfc-da5a3cb34950
Entering [CsSnapRequestor::AddVolumesToSnapshotSet]
CsSnapRequestor::AddVolumesToSnapshotSet() - Volume [\\?\VOLUME{0C022041-FF03-11DE-8B2F-806E6F6E6963}\] not supported by provider
Call AddVolumesToSnapshotSet(ids) [FAILED, throwing CV exception] - Code = 0x80320305, Description = Unknown error
Call CreateSnapshotSet() [FAILED, throwing CV exception] - Code = 0x80320305, Description = Unknown error
Entering [CsSnapRequestor::BackupComplete]
Entering [CsSnapRequestor::DeleteFolderMetadata]
CsSnapRequestor::DeleteFolderMetadata() - Not attempting to delete folder metadata
Entering [CsSnapRequestor::DeleteSnapshot]
CsSnapRequestor::DeleteSnapshot() - Shadow not created!
CsSnapRequestor::BackupComplete() - Deleted snapshot set
CsSnapRequestor::BackupComplete() - Aborting backup!
Entering [CsVssBackupComponents::Delete]
Entering [CsVssBackupComponents::FreeWriterMetadata]
CsVssBackupComponents::FreeWriterMetadata() - Free'd all writer metadata
FindWorkThreadWrapper::createVssSnap() - Not using writers for backup
FindWorkThreadWrapper::ProcessSysStateComp() - Failed to create a VSS snapshot and it is required to run a system state backup. Failing the scan

Microsoft Application or System Event Logs may display the following Event IDs:

Event IDs 12290 and 16387 are logged when system state backup fails on a Windows Server 2008-based computer.
Log Name: Application
Source: VSS
 

Event ID: 12290
Level: Warning
Description: Volume Shadow Copy Service warning: ASR writer Error 0x80070001. hr = 0x00000000.
Log Name: Application
Source: SPP
 

Event ID: 16387
Level: Error
Description: Shadow copy creation failed because of error reported by ASR Writer. More info: Incorrect function. (0x80070001).
 

Solution

Make sure that the drive that hosts the snapshots has sufficient space to post the new snapshot.

To resolve this issue,

  1. Verify if there is an Active, Primary Partition that is not an NTFS file system and does not have a Volume drive letter assigned.

    The following image is an example of an Active, Primary Partition that is not NTFS (New Technology File System).

    Run mountvol command from the Command Prompt at the administrator level. This will list the volumes available and their names.

    You can also view this list under the registry key located at HKLM\System\MountedDevices.

    Look for a data set without a matching\DosDevice\Volume drive letter. The following is an example of a data set without a matching DosDevice\Volume drive letter:

    \\?\Volume{0c022041-ff03-11de-8b2f-806e6f6e6963

    Review this with your Server Vendor or Server Image Provider to determine what volume is this for.

    If this does not resolve the issue, refer to Microsoft KB 968128 for more information.

WFS0016: Backups may fail to back up Windows System State "Event Logs"

Symptom

In Simpana software where VSS is used to backup the server, you may observe the following errors:

  • From CommCell Console:

Error Code: [82:128]
Description: The job is pending because: Network receive failed: The network connection was lost unexpectedly.
 

  • From Clbackup.log

    FsBackupTw::BackupComponentTemplate[Event Logs]() - Backing up [Event Logs]
    FsBackupTw::BackupComponentTemplate[Event Logs]() - ---->>>
    Entering [CDefaultWriterBackup::Backup]
    Entering [CDefaultWriterBackup::BackupWriter]
    Entering [CDefaultWriterBackup::GatherFlags]
    Entering [CDefaultWriterBackup::GatherWriterExcludes]
    CDefaultWriterBackup::GatherWriterExcludes() - Exclude file count [1] for writer [eee8c692-67ed-4250-8d86-390603070d00].
    CDefaultWriterBackup::BackupWriter() - Event Logs has 7 files, 0 logs, and 0 databases
    Entering [CWinBackup::IsFileExcluded]
    CWinBackup::IsFileExcluded() - Backup file [`g] matches writer exclude file pattern.
    CDefaultWriterBackup::BackupFileDesc() - Skipping filtered file [C:\WINDOWS\system32\config\AppEvent.Evt].
    Entering [CWinBackup::IsFileExcluded]
    CWinBackup::IsFileExcluded() - Backup file [ ^] matches writer exclude file pattern.
    CDefaultWriterBackup::BackupFileDesc() - Skipping filtered file [C:\WINDOWS\System32\Config\Internet Explorer.evt].
    CsTools::IsVolumeDevicePathForm() -Invalid arguments.
    ** CVSession::sendBytes(...) iSocketMember=860 send failed, err=[1][10093][Either the application has not called WSAStartup, or WSAStartup failed.].

    ** CVSession::sendMessageInt(...):
    - RemoteHost=ServerName.
    - RemoteProcess=AppMgrSvc.exe.
    - sendBytes() failed. Error=9000012.

    ** CVSession::disconnect (void)
    - RemoteHost=ServerName.
    - RemoteProcess=AppMgrSvc.exe.
    - sendMessageInt CVS_PM_DISCONNECT_REQ failed. Error=9000012.

    Error 9000012 is a Simpana Software Hex error which corresponds to the following error:

    Error Code 9:18
    []:Remote system []. The socket send failed. Error returned [=].

Solution

To resolve this issue, see Microsoft KB article 970219.

Additionally, refer to Microsoft Blog OpsMgr 2007: Backup programs may fail to backup the event log when the 64-bit agent is installed for more information.

WFS0017: Backup job pending with error: "File Scan failed to write to dirchange file"

Symptom

Backup job goes to a pending state with the following error:

Error Code 17:40
File Scan failed to write to dirchange file [...] . Possible reasons nonexistent path/file, hardware problems or full disk.

Solution

This issue is caused by a problem writing to the Job Results directory on the client computer.

  1. From the CommCell Console, navigate to Client Computers | <Client>.
  2. Click Advanced.
  3. Click Job Configuration tab. The Job Results Directory box contains the path where the job results files are listed.

    Make sure that the given path exists.

  4. Login to the Client Computer and verify if the current free space available is sufficient for hosting the Job Results Directory.

    For more information on how to determine the free space for Windows 2008 and Windows 2008 R2, see Microsoft KB article 2021748.

  5. Navigate to the path provided in the Job Results Directory box and check the size of the JobResults folder.

  6. Explore the options required to reduce the space or move JobResults folder to another location.

WFS0018: System state backup components fail

WMI

Symptom

Backup of WMI component may fail during system state backups and you may observe the following errors:

  • CommCell Console

Error Code 6:64
System State Backup of component [...] failed.

  • clbackup.log

<--- ADDING COMPONENT --->
5600 1474 07/08 09:45:31 {Job ID} Entering [CWinBackupJob::AddComponent]
5600 1474 07/08 09:45:31 {Job ID} Entering [CWmiBackup::GatherActiveStatus]

Resolution 1

To resolve the hang, run the following command to register the system performance libraries with WMI.

  • Windows 2008, Windows 2008 R2, type the following command:

    WINMGMT.EXE /RESYNCPERF

    For more information on this command, refer to Microsoft KB article 2554336.

  • Windows 2003, Windows XP, and Windows 2000, type the following command:

    WINMGMT.EXE /RESYNCPERF

    For more information on this command, see Microsoft article wmiadap.

  • Windows 2000 and NT, type the following command:

    winmgmt /resyncperf <winmgmt-service-process-id>

    For more information on this command, see Microsoft article winmgmt.

    For more information on how to build the WMI repository and Performance Counter Library values, see Rebuilding the WMI Repository and Microsoft KB article 300956.

Resolution 2

Filter the WMI data from the backup content if it is not necessary to be backed up.

For more information on how to filter system state components from a backup, see Filtering System State Components from a Backup.

RSM

Symptom

Backup of RSM component may fail during system state backups and you may observe the following errors:

  • CommCell Console

    Completed w/ one or more errors

  • Event Viewer

    System State Issue: At least one non-critical component has failed, see client logs for details.
    see clbackup.log for details.

    You may also receive the following error:

    Error Code 6:64:
    System State Backup of component [] failed.

  • clbackup.log

    <--- COMPONENT BACKUP REPORT --->
    Component Type Status
    --------- ---- ------
    RSM SYSTEM SERVICE FAILED

  • Reports

    To verify the failed component using reports, follow the steps given below:

    1. Open CommCell Console.
    2. On the CommCell Console menu bar, select the Reports tab and then click Summary.
    3. The Report Selection dialog box appears with Data Management selected by default.
    4. Click Selection tab. Select Failure Reason and Failed Objects.
    5. Click Run.

      Look for Files failed to backup to determine the failed files.

Solution

Disable RSM service or filter RSM component from the system state backup.

For more information on how to filter system state components from a backup, see Filtering System State Components from a Backup.

Content Indexing Catalogs

Symptom

Backup of Content Indexing Catalogs may fail during system state backups and you may observe the following errors:

  • CommCell Console

    Completed w/ one or more errors

  • Event Viewer

    System State Issue: At least one non-critical component has failed, see client logs for details.
    see clbackup.log for details.

    Error Code 6:64:
    System State Backup of component [] failed.

  • clbackup.log

    <--- COMPONENT BACKUP REPORT --->
    Component Type Status
    --------- ---- ------
    RSM SYSTEM SERVICE FAILED

  • Reports

    To verify the failed component using reports, follow the steps given below:

    1. Open CommCell Console.
    2. On the CommCell Console menu bar, select the Reports tab and then click Summary.
    3. The Report Selection dialog box appears with Data Management selected by default.
    4. Click Selection tab. Select Failure Reason and Failed Objects.
    5. Click Run.

      Look for Files failed to backup to determine the failed files.

Solution

To resolve this issue, restart the Indexing Service and make sure that the existing Catalogs are configured properly.

To verify indexing service configuration on computer, navigate to Computer Management | Services and Applications | Indexing Service.

NPS

Symptom

System state backup may fail with the following error:

Error Code 6:64 System State Backup of component [NPS Database] failed.

Resolution 1

To resolve this issue, make sure that the NPS (Network Policy Server) components are installed and actively used. If NPS is used actively, then you need to restore the database as it is a critical component of System State backup. For more information on critical and non-critical components on Windows operating system, see What are the critical and non-critical system state components on Windows operating system?

For more information, see Microsoft KB article 949470.

Resolution 2

Filter NPS (Network Policy Server) components from system state backup.

For more information on how to filter system state components from a backup, see Filtering System State Components from a Backup.

Additionally, you may also refer to the following references:

1-Touch Files

Backup of 1-Touch files may fail during system state backups and you may observe the following errors:

  • CommCell Console

    Completed w/ one or more errors

  • Event Viewer

    System State Issue: At least one non-critical component has failed, see client logs for details.
    see clbackup.log for details.

    Error Code 28:243
    Failed to collect disk, volume or partition information for 1-Touch Backup . 1-Touch recovery cannot be done. See client logs for details.

    Error Code 6:64:
    System State Backup of component [One Touch Files] failed.

    Error Code: 6:66
    System State Backup of component [One Touch Files;] failed[OneTouchError:Failed to backup 1-Touch file].

    Error Code 6:69:
    OneTouchError:Failed to backup 1-Touch file

  • clbackup.log

    <--- COMPONENT BACKUP REPORT --->
    Component Type Status
    --------- ---- ------
    One Touch Files ARCHIVE SUCCESS

  • Reports

    To verify the failed component using reports, follow the steps given below:

    1. Open CommCell Console.
    2. On the CommCell Console menu bar, select the Reports tab and then click Summary.
    3. The Report Selection dialog box appears with Data Management selected by default.
    4. Click Selection tab. Select Failure Reason and Failed Objects.
    5. Click Run.

      Look for Files failed to backup to determine the failed files.

Solution 1

Filter 1-Touch components from system state backup.

For more information on how to filter system state components from a backup, see Filtering System State Components from a Backup.

Solution 2

Make sure that the system state backup is not failing due to operating system error. The primary Windows component that causes 1-Touch failure is Windows VDS Basic Provider. The following is an example of an error that appears in Windows event logs:

Log Name: System
Source: VDS Basic Provider
Date: date time
Event ID: 1
Task Category: None
Level: Error
Keywords: Classic
User: N/A
Computer: computer name
Description: Unexpected failure. Error code: 490@01010004

Cluster Database

Symptom

System state backup of Cluster Database component may fail with the following errors:

  • CommCell Console

Error Code 6:64
System State Backup of component [Cluster Database] failed .

Error Code 28:105
System State Issue: At least one non critical component has failed, see clbackup.log for details.

  • clbackup.log

    <--- COMPONENT BACKUP REPORT --->
    Component Type Status
    --------- ---- ------
    Cluster Database SYSTEM STATE FAILED

  • Reports

    To verify the failed component using reports, follow the steps given below:

    1. Open CommCell Console.
    2. On the CommCell Console menu bar, select the Reports tab and then click Summary.
    3. The Report Selection dialog box appears with Data Management selected by default.
    4. Click Selection tab. Select Failure Reason and Failed Objects.
    5. Click Run.

      Look for Files failed to backup to determine the failed files.

Solution

Follow the steps given below to resolve this issue:

  1. Make sure that the Use VSS options is enabled in the General tab of the Subclient Properties dialog box prior to performing system state backup.
  2. Make sure that \\computername\ClusterLegacyBackup share is accessible from the node that is being backed up.
  3. Make sure that the correct computer name and is given in the UNC path and that it resolves to the correct IP address.
  4. If the client being backed up is a virtual machine, disable system state backups for that machine.
  5. Make sure that the Cluster services are up and running.

    For cluster computers operating on Windows 2003, you can use only Legacy method to backup the cluster component.

Additional Information

System state backups are capable of backing up both the active and passive nodes in a cluster database. When a system state backup is initiated, the active owner generates a dump of the Cluster database and places it on a network share. This dump is then retrieved and backed up by the node that performs the backup.

Cluster Database backup failures are caused mainly due to an inability to access these shares during the backup. You will observe the following error in clbackup.log:

<--- STARTING BACKUP OF Cluster Database --->
Cluster dump at [\\computername\ClusterLegacyBackup] and backup path [C:\Program Files\{Software Dir}\Simpana\iDataAgent\FileSystemAgent\2\123\clusterdb]
[::clusterBackup]: Name: NAME
::clusterBackup() - BackupClusterDatabase call failed with following error code [51]
<--- ENDING BACKUP OF Cluster Database - [FAILED] --->

Error codes 52, 53 and 1214 are all connectivity errors.

System Protected Files on Windows 2008 Server

Symptom

System state backup performed on Windows 2008 servers may fail to backup System Protected Files.

This error is caused by Access Denied Errors during backup for Cryptographic Service. You may observe the following errors:

  • CommCell Console

    Error Code 17:55:
    Failed to create a VSS snapshot and it is required in order to run a system state backup. Failing the scan.

  • clbackup.log

    FsBackupTw::BackupComponentTemplate[System Protected Files]() - FsBackupTw::BackupComponentTemplate[System Protected Files]() - Backing up [System Protected Files]
    FsBackupTw::BackupComponentTemplate[System Protected Files]() -


    Entering [CSystemProtectedFilesWriterBackup::Backup]
    Entering [CDefaultWriterBackup::Backup]
    Entering [CDefaultWriterBackup::BackupWriter]
    Entering [CDefaultWriterBackup::GatherFlags]
    Entering [CDefaultWriterBackup::GatherWriterExcludes]
    Entering [CsVssBackupComponents::CsVssBackupComponents]
    Entering [CsVssBackupComponents::Delete]
    Entering [CsVssBackupComponents::GetWriter]
    Entering [CsVssBackupComponents::GetWriter]
    Call GetWriter(writerGUID, pWriter) [FAILED, throwing CV exception] - Code=0x80004005, Description=E_FAIL
    CDefaultWriterBackup::GatherWriterExcludes() - vss.GetExcludeFiles(writerGuid, lExcFiles) - [FAILED] 0x80004005
    Entering [CsVssBackupComponents::~CsVssBackupComponents]
    Entering [CsVssBackupComponents::Delete]
    CsVssBackupComponents::Delete() - Assigned set, so not freeing metadata
    CDefaultWriterBackup::BackupWriter() - GatherWriterExcludes(args, writerGuid) - [FAILED] 0x80004005
    FsBackupTw::BackupComponentTemplate[System Protected Files]() - FsBackupTw::BackupComponentTemplate[System Protected Files]() - Failed to backup System Protected Files
    FsBackupTw::BackupComponentTemplate[System Protected Files]() -
    FsBackupTw::DoSystemStateBackup() - Critical error detected in component [System Protected Files]. Failing system state backup
    PrintReport() - Component Method Status

  • filescan.log

    Entering [CSystemProtectedFilesWriterBackup::Backup]
    Entering [CDefaultWriterBackup::Backup]
    Entering [CDefaultWriterBackup::BackupWriter]
    Entering [CDefaultWriterBackup::GatherFlags]
    Entering [CDefaultWriterBackup::GatherWriterExcludes]
    Entering [CsVssBackupComponents::CsVssBackupComponents]
    Entering [CsVssBackupComponents::Delete]
    Entering [CsVssBackupComponents::GetWriter]
    Entering [CsVssBackupComponents::GetWriter]
    Call GetWriter(writerGUID, pWriter) [FAILED, throwing CV exception] - Code=0x80004005, Description=E_FAIL
    CDefaultWriterBackup::GatherWriterExcludes() - vss.GetExcludeFiles(writerGuid, lExcFiles) - [FAILED] 0x80004005
    Entering [CsVssBackupComponents::~CsVssBackupComponents]
    Entering [CsVssBackupComponents::Delete]
    CsVssBackupComponents::Delete() - Assigned set, so not freeing metadata
    CDefaultWriterBackup::BackupWriter() - GatherWriterExcludes(args, writerGuid) - [FAILED] 0x80004005

  • Application or Event logs

    Product: Windows Operating System
    ID: 512
    Source: Microsoft-Windows-CAPI2
    Version: 6.0
    Symbolic Name: MSG_SYSTEMWRITER_INIT_FAILURE
    Message: The Cryptographic Services service failed to initialize the VSS backup "System Writer" object.%1.

    Product: Windows Operating System
    ID: 513
    Source: Microsoft-Windows-CAPI2
    Version: 6.0
    Symbolic Name: MSG_SYSTEMWRITER_ONIDENTITY_FAILURE
    Message: Cryptographic Services failed while processing the OnIdentity() call in the System Writer Object.%1.
     

    Log Name: Application
    Source: Microsoft-Windows-Backup
    Event ID: 517
    Level: Error
    Description:
    Backup started at '...date/time..' failed with following error code '2155348226' (System writer is not found in the backup.). Please rerun backup once issue is resolved.

Solution

These errors appear more than once if they are caused by a permission issue or when any of the system writers is missing from the list of writers.

These writers are required to backup System Protected Files.

Run the following command and review the output.

vssadmin list writers > c:\vssadmin.txt

For more information, see Microsoft article Vssadmin.

The output on Windows 2008 server would look similar to the one given below.

Writer Instance Id: {1754f85c-8bb4-4928-8b54-9536c984298c}
State: [1] Stable
Last error: No error

Writer name: 'Registry Writer'
Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
Writer Instance Id: {8c32dbe3-f944-4150-be3e-4bfc02b7295e}
State: [1] Stable
Last error: No error

Writer name: 'COM+ REGDB Writer'
Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
Writer Instance Id: {cd07fee3-a9f6-4424-b42e-2dad49d8a311}
State: [1] Stable
Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
Writer Instance Id: {ba399c41-f034-40c4-904d-9ae102c8bb1c}
State: [1] Stable
Last error: No error

Writer name: 'WMI Writer'
Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
Writer Instance Id: {199b8675-695c-4c62-9a55-5230b31d69bc}
State: [1] Stable
Last error: No error

Writer name: 'BITS Writer'
Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
Writer Instance Id: {6a8ac8d7-c598-4efd-913b-971148e89500}
State: [1] Stable
Last error: No error

For more information on system writers, see Microsoft KB article 2009272.

For Application or Event ID solutions, see Event ID 512 and Event ID 513.

For more information on missing system writers, see DPM 2007 Server 2008 System State Backup System Writer missing.

For more information on how VSS works, see How Volume Shadow Copy Works.

WFS0019: VSS errors

This is a collection of issues and resolutions of several VSS errors generated by Microsoft Windows operating system.

There are several VSS errors that generate the following errors:

  • CommCell Console

Error Code 17:55
Failed to create a VSS snapshot and it is required in order to run a system state backup. Failing the scan.

  • JobManager.log

    Scheduler Set pending cause [Failed to create a VSS snapshot and it is required in order to run a system state backup. Failing the scan.]::Client [server name] Application [FileScan] Message Id [285212727] RCID [0] ReservationId [0]. Level [0] flags [0] id [0] overwrite [0] append [0] CustId[0].
    Scheduler Phase [Failed] message received from [server name] Module [FileScan] Token [37376:4:1] restartPhase [0]
    JobSvr Obj Phase [4-Scan] for Backup Job Failed. Backup will continue with phase [Scan].

Symptom 1

System state backup of an active partition that is not NTFS formatted can result in backup failure.

Resolution 1

See Backup Failures on Windows 2008, Windows 2008 R2 or Windows 7 for more information.

Symptom 2

The following errors appear when the writers are in a bad or missing state.

  • Writer errors

    Writer name: 'System Writer'
    Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
    Writer Instance Id: {a9e9ad80-b17f-43fc-abde-693530722a5f}
    State: [9] Failed
    Last error: Timed out

    Writer name: 'WMI Writer'
    Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
    Writer Instance Id: {3ded1ea5-65c9-4e2b-bc0a-3064690a3eea}
    State: [9] Failed
    Last error: Timed out
     

    Writer name: 'Cluster Database'
    Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
    Writer Instance Id: {448f854c-ed09-439e-aeae-0e35e952ddab}
    State: [5] Waiting for completion
    Last error: No error

  • FileScan.log

    CsSnapRequestor::GatherWriterStatus() - Gathering writer status After DoSnapshotSet...
    CsSnapRequestor::GatherWriterStatus() - 10 writers responded
    STABLE status for writer 'Task Scheduler Writer'. code S_OK
    STABLE status for writer 'VSS Metadata Store Writer'. code S_OK
    STABLE status for writer 'Performance Counters Writer'. code S_OK
    FAILED_AT_FREEZE status for writer 'ASR Writer'. code VSS_E_WRITERERROR_TIMEOUT
    CsSnapRequestor::GatherWriterStatus() - Signaling bad state returned for writer [ASR Writer] engaged in backup.
    FAILED_AT_FREEZE status for writer 'System Writer'. code VSS_E_WRITERERROR_TIMEOUT
    CsSnapRequestor::GatherWriterStatus() - Signaling bad state returned for writer [System Writer] engaged in backup.
    FAILED_AT_FREEZE status for writer 'Shadow Copy Optimization Writer'. code VSS_E_WRITERERROR_TIMEOUT
    FAILED_AT_FREEZE status for writer 'COM+ REGDB Writer'. code VSS_E_WRITERERROR_TIMEOUT
    CsSnapRequestor::GatherWriterStatus() - Signaling bad state returned for writer [COM+ REGDB Writer] engaged in backup.
    WAITING_FOR_BACKUP_COMPLETE status for writer 'Registry Writer'. code S_OK
    WAITING_FOR_BACKUP_COMPLETE status for writer 'Cluster Database'. code S_OK
    FAILED_AT_FREEZE status for writer 'WMI Writer'. code VSS_E_WRITERERROR_TIMEOUT
    CsSnapRequestor::GatherWriterStatus() - Signaling bad state returned for writer [WMI Writer] engaged in backup.
    Call GatherWriterStatus("After DoSnapshotSet") [FAILED, throwing CV exception] - Code = 0x80004005, Description = E_FAIL
    Entering [CsSnapRequestor::DeleteFolderMetadata]
    CsSnapRequestor::DeleteFolderMetadata() - Not attempting to delete folder metadata
    Call DoShadowSet() [FAILED, throwing CV exception] -Code = 0x80004005, Description = E_FAIL
    Call CreateSnapshotSet() [FAILED, throwing CV exception] -Code = 0x80004005, Description = E_FAIL
    Entering [CsSnapRequestor::BackupComplete]

Resolution 2

Use writers.txt file to determine the writer causing the problem. For more information, see System Protected Files on Windows 2008 Server.

Symptom 3

This issue is mainly caused by a Diskshadow with a malformed path. For example, the following are registry keys available at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services where the file path is malformed.

  • Filelist.txt

- File List: Path = c:/progra~2/plixer~1\cgi-bin, Filespec = eventlogd.exe
- File List: Path = archivemanagerremoteinstaller, Filespec = remoteinstallerservice.exe
- File List: Path = c:/program files (x86)/ibm_ds/ibmstoragemanagerprofiler server\bin, Filespec = tomcat5.exe
- File List: Path = c:\windows\\systemroot\system32\drivers, Filespec = lsi_sas.sys
- File List: Path = \systemroot\c:\windows\system32, Filespec = svchost.exe

  • FileScan.log

    316 440 07/03 12:48:04 42346 CsSnapRequestor::AddVolumesToSnapshotSet() - Call m_vss->IsVolumeSupported(m_shadowProvider.GetProvider(), vol, &bSupported) [FAILED, throwing CV exception] - Code = 0x80042308, Description = VSS_E_OBJECT_NOT_FOUND
    316 440 07/03 12:48:04 42346 CsSnapRequestor::CreateSnapshotSet() - Call AddVolumesToSnapshotSet(ids) [FAILED, throwing CV exception] - Code = 0x80042308, Description = VSS_E_OBJECT_NOT_FOUND

Resolution 3

  1. Use shadow.txt to check for incorrect statements in the File Path section.
  2. Correct the malformed paths registry keys. Make sure to contact the Vendors owning the files as their installation has caused the malformed path.

For example:

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services
HKEY_LOCAL_MACHINE/SOFTWARE\Wow6432Node

Any path that has a / instead of \ does not conform to Windows LFS path standard. When the system writer identifies any malformed path, it cannot quiesce the file system for the VSS provider to take the snapshot.

Symptom 4

The following errors appear due to space issues with the Volume Shadow Service.

  • Server operating system Event Log File

    TYPE] Error [TIME] 2011/09/23 09:03:04 [SOURCE] volsnap [COMPUTER] server name [DESCRIPTION] When preparing a new volume shadow copy for volume ?:, the shadow copy storage on volume ??: could not be located in non-critical space. Consider using a shadow copy storage volume that does not have any shadow copies.

  • FileScan.log

    23924: [PID] [Thread] 09/23 09:03:05[JobID] CsVssAsync::WaitUntilDone() - Async status returned code = 0x8004231f, Description = Unknown status.
    23924: [PID] [Thread]09/23 09:03:05 [JobID] CsSnapRequestor::DoShadowSet() - Call asyncShadow.WaitUntilDone() [FAILED, throwing CV exception] - Code = 0x8004231f, Description = VSS_E_INSUFFICIENT_STORAGE
    23925: [PID] [Thread]09/23 09:03:05 [JobID] CsSnapRequestor::CreateSnapshotSet() - Call DoShadowSet() [FAILED, throwing CV exception] - Code = 0x8004231f, Description = VSS_E_INSUFFICIENT_STORAGE
    23926: [PID] [Thread]09/23 09:03:05 [JobID] CsSnapRequestor::Snap() - Call CreateSnapshotSet() [FAILED, throwing CV exception] - Code = 0x8004231f, Description = VSS_E_INSUFFICIENT_STORAGE
     

Resolution 4

Follow the steps given below to resolve this issue.

  1. Open My Computer on the client computer.
  2. Right-click on the <volume> and point to Properties.
  3. Click on Shadow Copies tab.
  4. Click Settings.
  5. In the Storage area, select Maximum Size as No Limit or the set the space limit required for the VSS copies. You can also move the holding section to another volume.

For Command Line, see Microsoft article Vssadmin add shadowstorage.

Symptom 5

When multiple VSS Providers are active on the server, you will notice the following in FileScan.log.

Description = VSS_E_UNEXPECTED_PROVIDER_ERROR
FindWorkThreadWrapper::w2kRecurse(4276): -Debug-: "C:\Users\addmtool\AppData\Local\History" is a junction point for "C:\Users\addmtool\AppData\Local\Microsoft\Windows\History".
Call CreateSnapshotSet() [FAILED, throwing CV exception] - Code = 0x8004230f, Description = VSS_E_UNEXPECTED_PROVIDER_ERROR
Resolution 5

Cause

Run the following commands from the Command Prompt as an administrator for Windows 2008 and Windows 2008 R2 client computers.

vssadmin list writers > c:\writers.txt

vssadmin list providers > c:\providers.txt

vssadmin list volumes > c:\volumes.txt

diskshadow /L c:\shadow.txt
list writers detailed
exit

For Windows 2003 run the following commands:

vssadmin list writers > c:\writers.txt

Mountvol > C:\moutvol.txt

VSSADMIN List SHADOWS > C:\Shadows.txt

VSSADMIN List SHADOWSTORAGE > C:\ShadowStorage.txt

VSSADMIN list providers > C:\providers.txt

Verify if the commands run from the command prompt display results similar to the following:

  • Volumes.txt

    Error: Unexpected failure: The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

  • Writers.txt

    Error: A Volume Shadow Copy Service component encountered an unexpected error.
    Check the Application event log for more information.
     

  • Providers.txt

Error: Unexpected failure: The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

  • DiskShadow.txt

    DISKSHADOW> list writers detailed
    COM call "(*vssObject)->InitializeForBackup" failed.
    The last operation failed.
    - Returned HRESULT: 80042302
    - Error text: VSS_E_UNEXPECTED
    DISKSHADOW> exit
    riters detailed

Resolution 5

These issues are caused mainly due to VSS Services not starting or running properly.

Follow the steps given below to resolve these issues:

  1. Make sure that the following services are not disabled.
    • Microsoft Software Shadow Copy Provider
    • Remote Procedure Call (RPC)
    • Volume Shadow Copy
  2. Check the Application and System event logs for VOLSNAP/VSS errors.
  3. Check the Providers installed using the following command:

    VSSADMIN LIST PROVIDERS

  4. Check the status of writers using the following command:

    VSSADMIN LIST WRITERS

    If you are still unable to resolve this issue, contact Microsoft.

    For more information, see Volume Shadow Copy Service, Volume Shadow Copy Service Operations for Windows 2008 R2, Volume Shadow Copy Service Operations for Windows 2008 R2.

Symptom 6

When multiple VSS Snap Providers are active on the server, you will notice the following in FileScan.log.

Entering [CsSnapRequestor::DeleteFolderMetadata]
CsSnapRequestor::DeleteFolderMetadata() - Skipping delete of folder metadata.
Call DoShadowSet() [FAILED, throwing CV exception] - Code = 0x80042306, Description = VSS_E_PROVIDER_VETO
Call CreateSnapshotSet() [FAILED, throwing CV exception] - Code = 0x80042306, Description = VSS_E_PROVIDER_VETO

When you run the VSSADMIN PROVIDERS LIST from the command prompt, the output would be similar to the following:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.
Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
Version: 1.0.0.7
Provider name: 'Data ONTAP VSS Hardware Provider'
Provider type: Hardware
Provider Id: {ddd3d232-a96f-4ac5-8f7b-250fd91fd102}
Version: 6. 3. 0. 4601

Resolution 6

nUseVSSSoftwareProvider additional settings must be added on the Client computer which has multiple VSS Providers on it. If the client machine has more than one iDataAgent installed on it, then this additional setting must be created multiple times under the corresponding agents.

For more information on this additional setting, see nUseVSSSoftwareProvider.

Additional information

You may refer to the following links for more information.

UFS0002: Error registering job with the Communications Service

Symptom

  • Backup fails with these error messages under the main error code 19:857.

19:210"Error registering job with the Communications Service."

  • Additional error messages seen from Events tab in the Backup Job Details window.

17:2"File system scan Failed"

  • Filescan.log file on the client will show this error:

Failed to translate "servername" to an IP address. Host ignored.
Find::impersonateUser()() - No user id found from the registry [VMName = ... ]. Trying getting it from the CS
::GetSubclientDir() - The Subclient directory is [d:\Program Files\Commvault Systems\Galaxy\iDataAgent\JobResults\CV_JobResults\iDataAgent\FileSystemAgent]
GetCommServerName() - CS name is [...]
Overiding CommServer name [server name] with registry value [registry server name]
JM Client CVBkpJobClient::init(): Initializing job object with token [...].
CVJobCtrlClient::registerJob() CVD replied with a non-zero error code [1] for the job registration message. Error = [Failed to get host name for client [...] for job [...]]
Init() - Error registering JOB with the Communications Service for JobID [...]
Init() - registerJob failed. Returning failure
Could not initialize job control.
BKP CALLED COMPLETE (PHASE Status::FAIL), ... . Token [...]

  • JobManager.log file on the CommServe will show this error:

Scheduler Set pending cause [Failed to get host name for client [...] for job [...]]::Client [...] Application [FileScan] Message Id [318767961] RCID [0] ReservationId [0]. Level [0] flags [0] id [0] overwrite [0] append [0] CustId[0].
Scheduler Ignored pending cause [Insufficient information to start the job. Possibly mismatched version with CommServe or corrupted install.]::Client [...] Application [FileScan] Message Id [285212706] RCID [0] Reservation Id [0]. Level [0] flags [0] id [0] overwrite [0] append [0] CustId[0].
Scheduler Phase [Failed] message received from [...] Module [FileScan] Token [...] restartPhase [0]
AppManager JMAppMgr::getValidateArchiveIndexPhase() failed. Returning false
JobSvr Obj Phase [4-Scan] for Backup Job Failed. Backup will continue with phase [Scan].

  • From the logs these additional error codes are part of the message:

In the Filescan.log file on the client

30:98 Failed to initialize Job Control.

Error message variations based on iDataAgent:

7:28 Could not initialize job control object.

39:37 Could not initialize job object.

In the JobManager.log file on the CommServe

17:34 Insufficient information to start the job. Possibly mismatched version with CommServe or corrupted install.

Error message variations based on iDataAgent/ MediaAgent/ or CommServe involved:

19:1033 Failed to get the host name.

19:104 Failed to get the target MediaAgent name for job [].

19:1102 Failed to get the MediaAgent name. Please verify that the MediaAgent version is compatible with the client version.

19:249 Failed to get the CommServe host name from the platform information.

  • In the EvMgrC.log file:

16:3 [] on computer [] failed to start; Failed to get local host name.

16:5 [] on computer [] failed to start. Host Name for this computer may not be configured properly.

  • For a restore job, you may receive this error in the clrestore.log file:

23:84 Restore: Wrong parameters for the job. Possibly mismatched version with CommServe or corrupted install.

  • When testing connectivity to the client computer with issues make sure to test name resolution in the following communications paths:
    • CommServe to Client, and from Client to CommServe
    • CommServe to MediaAgent, and from MediaAgent to CommServe
    • Client to MediaAgent ,and from MediaAgent to Client

The error is caused by DNS name resolutions not functioning properly in all directions. DNS or equivalent Lookup and Reverse Lookup must be functioning.

Local Windows Server Resolver Cache on each computer in the environment must be checked and Hosts files must be reviewed.

Resolution

Check the configuration in the CommCell Console for the CommServe, MediaAgent and the Client with issues. Different phases of the backup or restore operations require reviewing different communications paths. Also, check the registry entries on the CommServe, MediaAgent and the Client.

The example below will cover the CommServe and the client computers.

  1. From the CommCell Browser, right click the CommServe and click Properties. Check the CommCell Host Name.

  2. From the CommCell Browser, right click the Client computer and click Properties. Check the Host Name and CommServe Host Name.

  3. The registry on the CommServe (CS) will have two locations that show the CS Host Name when it also has a Client (CL) installed.

    For the CommServe Name used by the Clients other then the CommServe Client the registry entry for sCSHOSTNAME is located under \HKLM\Software\Commvault Systems\Galaxy\Instance001\CommServe

    For the CommServe with a Client (CL) File System iDataAgent installed, the name is hosted at this registry location.

    Registry entry for wDefaultHostName is located under \HKLM\Software\Commvault Systems\Common

  4. Now to get the information for the MediaAgent (MA) or the Client (CL), locate the MA or CL in the CommCell Browser window, right click, and then click  Properties. On the General tab, the following will be displayed.

    Client used as example here:

  5. Now once the MediaAgent (MA) and/ or Client (CL) CommCell information is confirmed, go to the MA and/or CL and review the registry entries for the Name.

    Client computer is used for example here.

    • To confirm the CL Host Name, check the registry on the CL for the entry wDefaultHostName located under \HKLM\Software\Commvault Systems\Common.

    • Now confirm the CommServe Hostname on the CL in the registry for the entry sCSHOSTNAME located under \HKLM\Software\Commvault Systems\Galaxy\Instance001\CommServe

    • For UNIX/ Linux based MA or CL, running the simpana status command will display the CommServe (CS) name as it appears in the /etc/CommvaultRegistry/Galaxy/Instance001/CommServe folder. The .properties file at this location contains the setting.

      [[root@lx5 ~]# simpana status
      --------------------------------- Instance001 ----------------[ General ]
      Version = 10(BUILD116) SP3a
      CommServe = ameba.id.loc
      Home Directory = /opt/simpana/Base
      Log Directory = /var/log/simpana/Log_Files
      Core Directory = /opt/simpana
      Temp Directory = /opt/simpana/Base/Temp
      Platform Type = 4
      Cvd Port Number = 8400
      EvMgrC Port Number = 8402
      [ Package ]
      1002/CVGxBase 10(BUILD116) = File System Core
      1101/CVGxIDA 10(BUILD116) = File System
      1112/CVGxDC 10(BUILD116) = Data Classification Enabler
      - Cache Root Directory = /opt/simpana/iDataAgent/jobResults/cvdccache
      1113/CVGxFSF 10(BUILD116) = File System Filter Driver
      [ Physical Machine/Cluster Groups ]
      Display Name = doclx5
      - Client Hostname = doclx5
      - Job Results Directory = /opt/simpana/iDataAgent/jobResults
      [root@lx5 ~]#

  6. Now that the CommServe Hostname and the Host Name are confirmed, first confirm DNS supports Lookup and Reverse Lookup for the names. If this is correct and communication works in all directions based on ping, ping -a and nslookup, retry the backup or restore operation.
    • If this does not resolve the issue then review the Name Management documentation to get the names matching in the environment.
    • If this does not resolve the issue then move to first using IP addresses between the CS, MA and CL instead of names.
    • If this fails to resolve the issue then use host file entries to populate the Resolver cache with the name translations.

      Enter in this format

      <IP Address> <Short Name> <FQDName>

Confirm the issue is resolved by running a backup or restore job and checking the Filescan.log, JobManager.log and clRestore.log files.

UFS0005: After an authentication failed error if a MediaAgent or a client cannot communicate with CommServe it will disable itself immediately

Symptom:

During Backup:

  • Error Code19:1327

Attempt start error: [...]

  • Error Code 9:90

Authentication failed for host [...]. Network password does not match.

  • Error Code 40:52

Failed to contact CommServe Services from the MediaAgent[...] during the start of the Data Transfer operation. Please check the network connectivity from this MediaAgent to the CommServe and make sure services are running on the CommServe.

  • Error Code 22:30

Could not establish connection to [...] on attempt [...].

  • Error Code 19:1131

Waiting for the services on the client [...] to come online.

  • Error Code 19:1597

Failed to start phase [...] on [...] due to network error [...]. Will attempt to restart. Please check if this product's services are running on the remote host.

  • Error Code 7:77

Unable to run [...] on the client. [CVSession::authenticateClient]:Remote system [FQDNservername]. Failed authentication returned from server.

  • Error Code 7:102

Unable to run [...] on the client.

  • Error Code 7:111

Unable to run [...] on the client. [...]

Error Code 9:36 [...]:Remote system [...]. Could not get the password from platform information or the database - authentication failed.

Error Code 9:37 [...]:Remote system [...]. Could not find the client password in the database - authentication failed.

Error Code 9:38 [...]:Remote system [...]. Failed authentication returned from server.

Error Code 9:39 [...]:Remote system [...]. Failed authentication returned from client.

Error Code 9:40 [...]:Remote system [...]. Unexpected message type received [...=...] - authentication failed.

Error Code 9:41 [...]:Remote system [...]. Could not encrypt challenge string when connecting - authentication failed.

Error Code 9:42 [...]:Remote system [...]. Password received from client did not match the database password for that client - authentication failed.

Error Code 9:89 Authentication failed for host [...]. Could not find the client entry in database.

Error Code 9:90 Authentication failed for host [...]. Network password does not match.

Error Code 9:91 CVSession Authentication failed for host [...]. Password is not available on the host.

During Restore:

  • Error Code 23:98

Seek failure happened

Additional Log Cuts:

  • From the JobManager.log file on the CommServe:

6956 2764 07/23 16:00:21 53303 Servant [---- SCHEDULED BACKUP REQUEST ----], taskid [84] Clnt[...] AppType[Windows File System][33] BkpSet[defaultBackupSet] SubClnt[default] BkpLevel[Incremental][2]
6956 27dc 07/23 16:00:24 53303 Scheduler Phase [4-Scan] could not start on [CLFQDNservername] due to Network Failure

  • From the CVD.log file on the Client:

8000 1f64 07/25 13:14:33 ### Unable to resolve remote host_Disabled_CSFQDNServerName to any IP family: [1][11001][No such host is known.]
8000 1f64 07/25 13:14:33 ### ** CVSsession::attach(ulPortArg):
- RemoteHost=_Disabled_CSFQDNServerName. - RemoteProcess=. Port =65535,65535
- getConnection() failed. Error=900004b.
8000 1f64 07/25 13:14:33 ### ** CVSession::getClientPassword ():
- RemoteHost=_Disabled_CSFQDNServerName.
- RemoteProcess=JobMgr.exe.
- commServerAttach (ePlatformTypeMember) failed. Error=900004b.
8000 1f64 07/25 13:14:33 ### ** CVSession::replyAttach (PlatformType)
- RemoteHost=CSFQDNServerName.
- RemoteProcess=JobMgr.exe.
- authenticateServer failed. Error=9000024.

Resolution:

If the nChatterFlag additional setting is set to 1, normal Communication and Connection testing will work properly but communication between MediaAgent or Client and the CommServe will continue to fail.

NOTE: The core issue is the MediaAgent or Client realized it was getting nowhere trying to talk (authenticate) to the CS and stops talking by setting the nChatterFlag and other registry key changes. The cause of this type of behavior is outlined above but may also be caused by transient or extended CVD communication issues between elements of the CommCell and the CommServe.

To fix the issue, check the additional settings on the CommCell Components affected.

On Windows:

  1. Check the nChatterFlag additional setting.
    • From the CommCell Browser, navigate to Client Computers.
    • Right-click the <Client> or <MediaAgent>, and then click Properties.
    • Click Advanced and then click Additional Settings tab.
    • Click Add.
    • In the Name box, type nChatterFlag.
    • The Category and Type fields are populated automatically.
    • In the Value box, type 0.

      This enables communication to the CommServe.

    • Click OK.
  2. Check the sCSHOSTNAME additional setting.
    • From the CommCell Browser, navigate to Client Computers.
    • Right-click the <Client> or <MediaAgent>, and then click Properties.
    • Click Advanced and then click Additional Settings tab.
    • Click Add.
    • In the Name box, type sCSHOSTNAME.
    • The Category and Type fields are populated automatically.
    • In the Value box, if the current string value has _Disabled_ appended at the front of the string value, remove it to allow communication to the CommServe.
    • Click OK.
  3. If there is a firewall between the MediaAgent or Client and the CommServe, check the nDISABLE_CVFWD additional setting.
    • From the CommCell Browser, navigate to Client Computers.
    • Right-click the <Client> or <MediaAgent>, and then click Properties.
    • Click Advanced and then click Additional Settings tab.
    • Click Add.
    • In the Name box, type nDISABLE_CVFWD.
    • The Category and Type fields are populated automatically.
    • In the Value box, type 0.

      This enables controlled connection to the CommServe.

    • Click OK.
  4. Use the SCM utility to restart Simpana services.

On UNIX:

It is recommended to stop Simpana services on the MediaAgent(s) and/or Client(s) before modifying additional settings.

  1. Check the nChatterFlag additional setting.
    • From the CommCell Browser, navigate to Client Computers.
    • Right-click the <Client> or <MediaAgent>, and then click Properties.
    • Click Advanced and then click Additional Settings tab.
    • Click Add.
    • In the Name box, type nChatterFlag.
    • The Category and Type fields are populated automatically.
    • In the Value box, type 0.

      This enables communication to the CommServe.

    • Click OK.
  2. Check the sCSHOSTNAME additional setting.
    • From the CommCell Browser, navigate to Client Computers.
    • Right-click the <Client> or <MediaAgent>, and then click Properties.
    • Click Advanced and then click Additional Settings tab.
    • Click Add.
    • In the Name box, type sCSHOSTNAME.
    • The Category and Type fields are populated automatically.
    • In the Value box, if the current string value has _Disabled_ appended at the front of the string value, remove it to allow communication to the CommServe.
    • Click OK.
  3. If there is a firewall between the MediaAgent or Client and the CommServe, check the nDISABLE_CVFWD additional setting.
    • From the CommCell Browser, navigate to Client Computers.
    • Right-click the <Client> or <MediaAgent>, and then click Properties.
    • Click Advanced and then click Additional Settings tab.
    • Click Add.
    • In the Name box, type nDISABLE_CVFWD.
    • The Category and Type fields are populated automatically.
    • In the Value box, type 0.

      This enables controlled connection to the CommServe.

    • Click OK.
  4. Restart Simpana services.
  5. Test Backups and Restores to confirm data protection has been restored.

Special Case Scenario:

This resolution does not work if the pNET password on the MediaAgent or Client and the entry in the CommServe database do not match.

  1. Check the CVD.log on the MediaAgent or Client for these entries:

    3756 d98 04/25 16:17:23 ### [ClientSessionWrapper::connectToCS] Error connecting to svc [AppManager network access] on CS. [150994982-[CVSession::authenticateClient]:Remote system [servernameFQDN]. Failed authentication returned from server.]
    3756 d98 04/25 16:17:23 ### ClientInterfaceNet::lookup() - [ClientSessionWrapper::connectToCS] Error connecting to svc [AppManager network access] on CS. [150994982-[CVSession::authenticateClient]:Remote system [servernameFQDN]. Failed authentication returned from server.]
    3756 d98 04/25 16:17:23 ### ::UpdateClientInfo() - Failed to do the client lookup for CLHostName
    3756 d98 04/25 16:17:23 ### ::UpdateClientInfo() - Session connect failed with error code 150994982 - [CVSession::authenticateClient]:Remote system [servernameFQDN]. Failed authentication returned from server., Generic error 150995034
    3756 d98 04/25 16:17:23 ### ::UpdateClientInfo() - Setting nChaterFlag to 1 to stop all communications
    3756 d98 04/25 16:17:23 ### ::UpdateClientInfo() - Setting nDISABLE_CVFWD to 1 to stop all communications
    3756 d98 04/25 16:17:23 ### ::UpdateClientInfo() - CS Host Name set to _Disabled_servernameFQDN to avoid any contact from this client to CS

  1. Check for a matching entry with same Date/Time stamp in the CVD.log file on the CommServe:

    4532 92c 04/25 15:08:33 ### EvSANVolMgr::DiscoverFLRSupportedApps() - CVSession::attach() to [ipaddress*CLHostName*8400*8402] returned error 9000026
    4532 92c 04/25 15:08:33 ### ::sendResponse() - FAILED [Failed to connect to remote host [ipaddress*CLHostName*8400*8402].]
    3196 bc0 04/25 16:17:23 ### [CVD ] ** CVD_CVSESSION_ERROR: RemoteHost=ipaddress? Error replying to attach on socket: 900002a=[CVSession::authenticateServer]:Remote system [ipaddress]. Password received from client did not match the database password for that client - authentication failed.

WFS:0032 Skip deleted or missing files during backup

Symptom

If certain files are found during scan phase and are missing during backup, backup of these files would fail with the following errors. Also, these files would be counted as failed objects.

ERROR_FILE_NOT_FOUND

ERROR_PATH_NOT_FOUND

Resolution

Enable the nSkipFileNotFoundOnBackup additional settings to resolve this issue. When this key is enabled, backup job skips the missed files and these files are not counted as failed objects. Make a note that this key is applicable only for missing files and not for missing directories. Any missing directory found would be counted as an failed object.

Follow the steps given below to enable this additional setting:

  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click the <Client> on which you want to add the additional setting, and then click Properties.
  3. Click Advanced.
  4. Click the Additional Settings tab.
  5. Click Add.
  6. In the Name field, type nSkipFileNotFoundOnBackup.
  7. The Category and Type fields are populated automatically.
  8. In the Value field, type 1.
  9. Click OK.

WFS0033: Contents are not backed up although the backup completes successfully

Symptom

When a backup operation is performed on Windows 32-bit computers, some content may not get backed up although the backup job completes successfully with no errors.

Resolution

This is a Microsoft Windows 2003/Windows XP known issue. Open and Close the registry key located at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\BackupRestore\FilesNotToBackup\DRM.

For more information, see Microsoft KB 927389 and ANS1009W DRM FilesNotToBackup RC 13.

WFS0034: Operating system service failures

Symptom

Files backed up using VSS services and restored using Windows File System iDataAgent might be corrupted.

Cause

Simpana leverages Microsoft Windows Volume Shadow Copy services (VSS) to generate snapshots of the data residing on the client computer. If this snapshot is corrupted, data might also get corrupted.

Resolution

Contact Microsoft to obtain hot fixes for this issue. For more information, see Microsoft KB article 2748349.

Additional information

For additional information, see Potential Sparse File Corruption on Windows operating system.

WFS0035: Backup failure of cluster computers

Symptom

After you perform a disaster recovery of cluster computers, backup of the virtual client that manages the cluster computers will fail.

Cause

The virtual client fails to connect to the cluster computers after you perform the System State recovery.

Solution

  1. Right-click the virtual client that manages the cluster computers and point to Properties.
  2. In the Client Computer Properties dialog box, click Advanced.
  3. In the Advanced Client Properties dialog box, click Cluster Group Configuration tab.
  4. Enable the Force Sync configuration on remote nodes checkbox.
  5. Click OK.
  6. Restart the services on both nodes of the cluster.

WFS0036: Incremental Backups are Larger than Expected

Symptom

When you perform an incremental backup of files that reside in a volume configured to use Microsoft deduplication, you may notice that the size of the incremental backup is the same as that of a Full Backup.

Cause

The large size of the incremental backups may be due to the scan method that is used.

Solution

If your subclient content consists of files that reside in a volume configured to use Microsoft deduplication, use the Recursive Scan method to back up your files. Clear the Check Archive Bit check box before you perform your backup.

For more information, see Configuring Scan method for Detecting Changed Files.

WFS0037: Backup Using Post-Backup Process Fails because of Using a Different MediaAgent

Symptom

A backup using post-process scripting fails because the Archive Index phase uses a MediaAgent in a GridStor setup that is different from the one used during the backup phase.

Cause

If the customer uses a post-backup script on a 10.0 system that was upgraded from 9.0, the MediaAgent used during the backup is released when the backup phase completes.

Solution

Create and/or enable the HoldReservationDuringPostBkp Additional Setting. This ensures that the MediaAgent is "sticky"; that is, it is retained for the Archive Index phase rather than being released at the end of the backup phase. Follow the procedure in Add or Modify an Additional Setting, using a Value of 1 and confirming that the Enable option is selected.