> Documentation > Backup and Restore Agents > Backup Agents > Microsoft Windows > Troubleshooting

Backup Troubleshooting

Knowledge Base Articles

WFS0001: Backup failure while backing up large files or system volumes

Cause

When backing up large files and/or system volumes, the Windows memory pool resources may run low and the backup may fail with errors reported in the Windows Event Viewer and the iDataAgent logs.

Solution

Review the Microsoft documentation for increasing the maximum memory pool usage and paged memory pool size.

WFS0002: Backup goes to pending state while backing up an unformatted volume and mount points

Symptom

The Event Viewer displays following message: Cannot scan <path>

Solution

Add the NSkipDirErrors additional setting and set the value at 1 (or any value other than 0)

WFS0003: Backup goes to pending state for Polyserve (Non-Microsoft clusters)

Symptom

The backup job for a Polyserve goes to pending state

Solution

Exclude the following folders from the backup and anti-virus scan:

For backup

c:\program files\polyserve\matrixserver\conf\mounts

For anti virus scans

c:\program files\polyserve\matrixserver\conf

<Commvault install directory>/Base

During Disaster Recovery, open up the Polyserve Configuration Utility and export the configuration to the node that was rebuilt. This will ensure complete synchronization prior to introducing a rebuilt node again into the matrix.

WFS0005: Completed with one or more errors

WFS0006: Full backup of Lync server when Incremental or Differential backup is performed

When a subclient is configured for a Lync server, always a full backup of the data and settings is performed, irrespective of the backup type selected for the subclient.

WFS0007: Lync server data and settings are not backed up

The Lync server must be running when you perform backup of the subclient configured for the Lync server.

WFS0009: Failure of File System scan

Cause

Backup operation will fail at the scan phase if you have OPSWAT Secure RAM disk software installed on your computer that is being backed up. This is because the virtual disk created by the OPSWAT Secure RAM disk software is not being identified as a RAM disk thereby causing the file system scan to fail.

Solution 1

The virtual disk created by the OPSWAT Secure RAM disk software must be excluded from the subclient content by adding the virtual drive to the list of filters. For more information on adding content to filters, see Configuring Filters for Backups.

Solution 2

Specify the drives that need to be backed up manually in the subclient content.

WFS0010: Synthetic Full backup failed

If an absolute path is given as the subclient content and the path does not contain any files/folders, then the full backup would complete successfully but the subsequent synthetic full backup would fail.

WFS0011: Files appear corrupted

Backup files that are created by using the Windows Volume Shadow Copy feature are corrupted in Windows Server 2003, Windows Vista, Windows Server 2008, Windows 7 and Windows Server 2008 R2.

This is a Microsoft known issue, see Microsoft KB 2748349 for more information.

WFS0012: Potential sparse file corruption on Windows operating system

Symptom

Data corruption occurs on a computer operating on Windows 7 or Windows 2008 R2 when a sparse file undergoes random write stress, and the data in the sparse file is overwritten with 64 KB of zeros. This happens due to a rare condition in the NTFS file system driver. This behavior causes data corruption under heavy write stress for sparse files.

You may receive the following error codes:

Error Code 9:30
 [...]:Remote system [...]. The socket is in an invalid state [...] for desired action
Error Code 13:127 
 Error occurred while processing chunk [...] in media [...] for storage policy [...] copy [...]: Backup job [ ... ]. Data read from media appears to be corrupted.
Error Code 62:469 
 Data read from media appears to be corrupted

The Tag Header for all logs will always return a NULL or =Blankspace value as follows:

  • For a restore or a synthetic full job, the following errors would appear in CVD.log on the MediaAgent

    8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] **ERROR** DataMoverSingleInstancer::ReadTagHeader: Failed to convert the tag header to Host byte order. Tag header got = 
 8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] The current Read offset into the chunk Physical 11602 Logical 0 
 8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] Metadata file [F:\Backup\CV_MAGNETIC\V_1211796\CHUNK_5018060\CHUNK_META_DATA_5018060].
 8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] Data file [F:\Backup\CV_MAGNETIC\V_1211796\CHUNK_5018060\SFILE_CONTAINER_001]. Version [2], SFile Number [1], Last Seek Offset [904877]
 8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] Current data file offset [904973]

  • For an Auxiliary Copy job, the following errors would appear in AuxCopy.log on the MediaAgent

    8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] **ERROR** DataMoverSingleInstancer::ReadTagHeader: Failed to convert the tag header to Host byte order. Tag header got = 
 8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] The current Read offset into the chunk Physical 11602 Logical 0 
 8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] Metadata file [F:\Backup\CV_MAGNETIC\V_1211796\CHUNK_5018060\CHUNK_META_DATA_5018060].
 8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] Data file [F:\Backup\CV_MAGNETIC\V_1211796\CHUNK_5018060\SFILE_CONTAINER_001]. Version [2], SFile Number [1], Last Seek Offset [904877]
 8484 2368 07/12 14:23:57 549154 7225349-7257065 [DM_BASE ] Current data file offset [904973]
Additional logs available on the MediaAgent

  • CVD.log

    1896 164 07/25 08:53:02 1007062 #--1 [DM_BASE ] Opening the Chunk =6322105, ArchFileId = 1008183, FileMarker=623, ArchFilePhysSizeInChunk=2146389541 VolId=744525
 1896 164 07/25 08:53:02 1007062 #--1 [MEDIAFS ] Magnetic Mount Path :- KBRemaning 560559808, Minimum space MB to be left [2048], maximum space MB to be filled [-1]
 1896 164 07/25 08:53:02 1007062 #--1 [MEDIAFS ] RealMagneticFS Opened file
 1896 160c 07/25 08:53:16 1007062 [UNCOMPRESS ] #UNCOMPRESSION ERROR! rc = -3 BN = 0 Scheme = 0 InputSize: 62974 UncompressedSize: 63392 ExpectedSize: 63392
 1896 164 07/25 08:53:26 ####### [CVD ] STOP_PL Control Message Received 0 (Pipeline Thread self-terminated)
 1896 1408 07/25 08:53:26 1007062 [DECRYPT ] STOP_PL while allocating a buffer
 1896 164 07/25 08:53:26 1007062 [FSRESTHEAD ] [FsRestoreHead::stopPipeline]: Received STOP_PL control message. Bailing out...
    1896 1df8 08/10 08:33:35 #######[ NULL] invalid tag header 1896 1df8 08/10 08:33:35 1016869 #--1 [DM_BASE ] ERROR DataMoverBase::Read: Failed to convert the tag header to Host byte order. Tag header got = 1896 1df8 08/10 08:33:35 1016869 #--1 [DM_BASE ] The current Read offset into the chunk Physical 1077840646652 Logical 1183781091958 1896 1df8 08/10 08:33:35 1016869 [DM_READER ] DataReader::Read: Failed to READ data from ARCHIVE FILE [1010462] COPY [134]. PHYSICAL LEN = [96] LOGICAL LEN =[0]. TAGHEADER = [1]
 1896 1df8 08/10 08:33:35 1016869 [FSRESTHEAD ] Encountered error when reading data from the media.

  • FSIndexedRestore.log 

    3164 1250 07/25 08:53:32 1007062 CVRestoreFiles::restoreFile() - Sending FSR_MSG_OPEN_ARCHIVE (2,18,1009296)
 3164 1250 07/25 08:53:32 1007062 CVRestoreFiles::sendMsgToClient() - CVSession::sendMessage() returned error [900001e=[CVSession::sendMessage]: Remote system [...]. The socket is in an invalid state [CVS_STATE_INIT] for desired action.]
 3164 1250 07/25 08:53:33 1007062 RSTCALLBACK: error restoring '\D:\Users\...\My Documents\DOCUMENTS\documents for publication\filename' afile id (2,18,1009296) offset hi 0 lo 1173355286
 3164 1250 07/25 08:53:33 1007062 RSTCALLBACK: cannot restore node 1571547 version 0 afile 352 offset 1173355286
 3164 1250 07/25 08:53:33 1007062 0 RESTORE: ERROR: Restore Failed for ...\Windows File System\defaultBackupSet\D Users(CommCell:AppNum).
 3164 1250 07/25 08:53:33 1007062 0 RESTORE: Source path [\] ([1] of [1]).
Solution

We recommend you to obtain the Microsoft hot fix for this issue. Refer to Microsoft KB 270881 to obtain the hotfix.

WINFS:0013 Incremental Backups Are of the Same Size As That of a Full Backup

Symptom

If you have installed Symantec Antivirus software on your client computer, when you perform an incremental backup followed by a full backup, the size of the incremental backup is the same as that of the full backup. Also, you may notice that a large amount of data is backed up.

Cause

This issue is due to the Symantec Antivirus software installed on your client computer. After Symantec Endpoint Protection performs a manual or scheduled scan, the access or modified time of the file is changed. As a result, the files are backed up again.

Solution

Create the NoFileMod setting as follows:

  • On a 32-bit version of a client computer that has Symantec Endpoint Protection installed on it, create the following DWORD value and set it to 1. HKEY_LOCAL_MACHINE\SOFTWARE\Symantec\Symantec Endpoint Protection\AV\NoFileMod

  • On a 64-bit version of a client computer that has Symantec Endpoint Protection installed on it, create the following DWORD value and set it to 1. HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Symantec\Symantec Endpoint Protection\AV\NoFileMod

For more information, see Symantec Endpoint Protection scan causes incremental backups to perform full backups.

WINFS0014: Data protection error: [Err:1450]"Insufficient system resources exist to complete the requested service"

Symptom

Data protection jobs may run slowly, fail, or fail to backup some large file with the following errors.

  • From clbackup.log on the client server.

    Error [1450] during BackupRead. Err:[1450] Insufficient system resources exist to complete the requested service.
    ERROR 1130: Not enough server storage is available to process this command.

  • Operating system Application Event Logs (When you use a third party backup software that uses NT Backup API)

    ERROR 1450: Insufficient system resources exist to complete the requested service.
 ERROR 1450: / hex 0x5aa ERROR_NO_SYSTEM_RESOURCES
 Operating system error 1450 Insufficient system resources exist to complete the requested service.
 Write on "device" failed, status = 1450
    ERROR 1130: Not enough server storage is available to process this command.
 ERROR 1130 / hex 0x46a ERROR_NOT_ENOUGH_SERVER_MEMORY
 Backup or restore operation terminating abnormally.
Solution

This is a Microsoft known issue. Refer to Microsoft KB article 304101 for more information.

Additionally, you can also look at Microsoft KB articles 2002606 and 967351 for more details.

WINFS0015: Backup failures on Windows 2008, Windows 2008 R2 or Windows 7

Symptom

The following errors may be generated during the backup operation:

  • From CommCell Console Event Viewer or Job Details dialog box:

    Error Code 17:55: 
 Failed to create a VSS snapshot and it is required in order to run a system state backup. Failing the scan. 
 FileScan Error:Cannot scan [\]

  • From the FileScan.log file on the Client server:

    CsVolumeList::Add() - VolGUID path (\\?\Volume{0c022041-ff03-11de-8b2f-806e6f6e6963}\Boot) too long. Truncating.
 GetVolumeMountPointsForVolumeGUID() - Found no path matching vol name [\\?\Volume{0c022041-ff03-11de-8b2f-806e6f6e6963}\].
 CsVolumeList::Add() - Not able to get volume mount points for vol [\\?\Volume{0c022041-ff03-11de-8b2f-806e6f6e6963}].
 CsVolumeList::Add() - Added [\\?\VOLUME{0C022041-FF03-11DE-8B2F-806E6F6E6963}\] to volume list

Once the FileScan.log shows the above error, then system state VSS snap is started for the file scan and the following error message is seen when the files scan or backup fails.

The cause for the failure is that VSS is unable to create a SNAPSHOT of the volume identified.

You may observe the following in the logs:

CsSnapRequestor::StartSnapshotSet() - Starting a new Snapshot Set
 Entering [CsVssBackupComponents::StartShadowSet]
 CsSnapRequestor::StartSnapshotSet() - Created shadow set e4875fdb-5caa-48ed-abfc-da5a3cb34950
 Entering [CsSnapRequestor::AddVolumesToSnapshotSet]
 CsSnapRequestor::AddVolumesToSnapshotSet() - Volume [\\?\VOLUME{0C022041-FF03-11DE-8B2F-806E6F6E6963}\] not supported by provider
 Call AddVolumesToSnapshotSet(ids) [FAILED, throwing CV exception] - Code = 0x80320305, Description = Unknown error
 Call CreateSnapshotSet() [FAILED, throwing CV exception] - Code = 0x80320305, Description = Unknown error
 Entering [CsSnapRequestor::BackupComplete]
 Entering [CsSnapRequestor::DeleteFolderMetadata]
 CsSnapRequestor::DeleteFolderMetadata() - Not attempting to delete folder metadata
 Entering [CsSnapRequestor::DeleteSnapshot]
 CsSnapRequestor::DeleteSnapshot() - Shadow not created!
 CsSnapRequestor::BackupComplete() - Deleted snapshot set
 CsSnapRequestor::BackupComplete() - Aborting backup!
 Entering [CsVssBackupComponents::Delete]
 Entering [CsVssBackupComponents::FreeWriterMetadata]
 CsVssBackupComponents::FreeWriterMetadata() - Free'd all writer metadata
 FindWorkThreadWrapper::createVssSnap() - Not using writers for backup
 FindWorkThreadWrapper::ProcessSysStateComp() - Failed to create a VSS snapshot and it is required to run a system state backup. Failing the scan

Microsoft Application or System Event Logs may display the following Event IDs:

Event IDs 12290 and 16387 are logged when system state backup fails on a Windows Server 2008-based computer. 
 Log Name: Application 
 Source: VSS
Event ID: 12290 
 Level: Warning 
 Description: Volume Shadow Copy Service warning: ASR writer Error 0x80070001. hr = 0x00000000.
 Log Name: Application 
 Source: SPP
Event ID: 16387 
 Level: Error 
 Description: Shadow copy creation failed because of error reported by ASR Writer. More info: Incorrect function. (0x80070001).
Solution

Make sure that the drive that hosts the snapshots has sufficient space to post the new snapshot.

To resolve this issue,

  1. Verify if there is an Active, Primary Partition that is not an NTFS file system and does not have a Volume drive letter assigned.

    unallocated_ntfs1

    The following image is an example of an Active, Primary Partition that is not NTFS (New Technology File System).

    unallocated_ntfs2

    Run mountvol command from the Command Prompt at the administrator level. This will list the volumes available and their names.

    You can also view this list under the registry key located at HKLM\System\MountedDevices.

    Look for a data set without a matching\DosDevice\Volume drive letter. The following is an example of a data set without a matching DosDevice\Volume drive letter: 

    \\?\Volume{0c022041-ff03-11de-8b2f-806e6f6e6963

    Review this with your Server Vendor or Server Image Provider to determine what volume is this for.

    If this does not resolve the issue, refer to Microsoft KB 968128 for more information.

WFS0017: Backup job pending with error: "File Scan failed to write to dirchange file"

Symptom

Backup job goes to a pending state with the following error:

Error Code 17:40 
 File Scan failed to write to dirchange file [...] . Possible reasons nonexistent path/file, hardware problems or full disk.
Solution

This issue is caused by a problem writing to the Job Results directory on the client computer.

  1. From the CommCell Console, navigate to Client Computers | <Client>.

  2. Click Advanced.

  3. Click Job Configuration tab. The Job Results Directory box contains the path where the job results files are listed.

    Make sure that the given path exists.

    job_results_directory

  4. Login to the Client Computer and verify if the current free space available is sufficient for hosting the Job Results Directory.

    For more information on how to determine the free space for Windows 2008 and Windows 2008 R2, see Microsoft KB article 2021748.

    job_results_directory1

  5. Navigate to the path provided in the Job Results Directory box and check the size of the JobResults folder.

    job_results_directory2

  6. Explore the options required to reduce the space or move JobResults folder to another location.

WFS0039: System state backup components fail

WMI
Symptom

Backup of WMI component may fail during system state backups and you may observe the following errors:

  • CommCell Console

    Error Code 6:64 
 System State Backup of component [...] failed.

  • clbackup.log

    <--- ADDING COMPONENT ---> 
 5600 1474 07/08 09:45:31 {Job ID} Entering [CWinBackupJob::AddComponent] 
 5600 1474 07/08 09:45:31 {Job ID} Entering [CWmiBackup::GatherActiveStatus]
Resolution 1

To resolve the hang, run the following command to register the system performance libraries with WMI.

  • Windows 2008, Windows 2008 R2, type the following command: 

    WINMGMT.EXE /RESYNCPERF

    For more information on this command, refer to Microsoft KB article 2554336.

  • Windows 2003, Windows XP, and Windows 2000, type the following command: 

    WINMGMT.EXE /RESYNCPERF

    For more information on this command, see Microsoft article wmiadap.

  • Windows 2000 and NT, type the following command: 

    winmgmt /resyncperf <winmgmt-service-process-id>

    For more information on this command, see Microsoft article winmgmt.

    For more information on how to build the WMI repository and Performance Counter Library values, see Rebuilding the WMI Repository and Microsoft KB article 300956.

Resolution 2

Filter the WMI data from the backup content if it is not necessary to be backed up.

For more information on how to filter system state components from a backup, see Configure Filters for Windows File System.

RSM
Symptom

Backup of RSM component may fail during system state backups and you may observe the following errors:

  • CommCell Console 

    Completed w/ one or more errors

  • Event Viewer 

    System State Issue: At least one non-critical component has failed, see client logs for details. 
 see clbackup.log for details.

    You may also receive the following error:

    Error Code 6:64:
 System State Backup of component [] failed.

  • clbackup.log 

    <--- COMPONENT BACKUP REPORT ---> 
 Component Type Status 
 --------- ---- ------ 
 RSM SYSTEM SERVICE FAILED

  • Reports

    To verify the failed component using reports, follow the steps given below:

    1. Open CommCell Console.

    2. On the CommCell Console menu bar, select the Reports tab and then click Summary.

    3. The Report Selection dialog box appears with Data Management selected by default.

    4. Click Selection tab. Select Failure Reason and Failed Objects.

    5. Click Run.

      Look for Files failed to backup to determine the failed files.

Solution

Disable RSM service or filter RSM component from the system state backup.

For more information on how to filter system state components from a backup, see Configure Filters for Windows File System.

NPS
Symptom

System state backup may fail with the following error:

Error Code 6:64 System State Backup of component [NPS Database] failed.
Resolution 1

To resolve this issue, make sure that the NPS (Network Policy Server) components are installed and actively used. If NPS is used actively, then you need to restore the database as it is a critical component of System State backup. For more information on critical and non-critical components on Windows operating system, see Critical and Non-Critical System State Components for Windows File System.

Resolution 2

Filter NPS (Network Policy Server) components from system state backup.

For more information on how to filter system state components from a backup, see Configure Filters for Windows File System.

Additionally, you may also refer to the following references:

1-Touch Files

Backup of 1-Touch files may fail during system state backups and you may observe the following errors:

  • CommCell Console 

    Completed w/ one or more errors

  • Event Viewer 

    System State Issue: At least one non-critical component has failed, see client logs for details. 
 see clbackup.log for details.
    Error Code 28:243
 Failed to collect disk, volume or partition information for 1-Touch Backup . 1-Touch recovery cannot be done. See client logs for details.
    Error Code 6:64:
 System State Backup of component [One Touch Files] failed.
    Error Code: 6:66
 System State Backup of component [One Touch Files;] failed[OneTouchError:Failed to backup 1-Touch file].
    Error Code 6:69:
 OneTouchError:Failed to backup 1-Touch file

  • clbackup.log 

    <--- COMPONENT BACKUP REPORT ---> 
 Component Type Status 
 --------- ---- ------ 
 One Touch Files ARCHIVE SUCCESS

  • Reports

    To verify the failed component using reports, follow the steps given below:

    1. Open CommCell Console.

    2. On the CommCell Console menu bar, select the Reports tab and then click Summary.

    3. The Report Selection dialog box appears with Data Management selected by default.

    4. Click Selection tab. Select Failure Reason and Failed Objects.

    5. Click Run.

      Look for Files failed to backup to determine the failed files.

Solution 2

Make sure that the system state backup is not failing due to operating system error. The primary Windows component that causes 1-Touch failure is Windows VDS Basic Provider. The following is an example of an error that appears in Windows event logs:

Log Name: System 
 Source: VDS Basic Provider 
 Date: date time
 Event ID: 1 
 Task Category: None 
 Level: Error 
 Keywords: Classic 
 User: N/A 
 Computer: computer name
 Description: Unexpected failure. Error code: 490@01010004
Workaround

Filter 1-Touch components from system state backup.

For more information on how to filter system state components from a backup, see Configure Filters for Windows File System.

Important:

  • Excluding 1-Touch components may affect the 1-Touch restore of the computer.

  • Filter 1-Touch components at the client level. Do not apply filters to global filters.

Cluster Database
Symptom

System state backup of Cluster Database component may fail with the following errors:

  • CommCell Console

    Error Code 6:64
 System State Backup of component [Cluster Database] failed .
    Error Code 28:105
 System State Issue: At least one non critical component has failed, see clbackup.log for details.

  • clbackup.log 

    <--- COMPONENT BACKUP REPORT ---> 
 Component Type Status 
 --------- ---- ------ 
 Cluster Database SYSTEM STATE FAILED

  • Reports

    To verify the failed component using reports, follow the steps given below:

    1. Open CommCell Console.

    2. On the CommCell Console menu bar, select the Reports tab and then click Summary.

    3. The Report Selection dialog box appears with Data Management selected by default.

    4. Click Selection tab. Select Failure Reason and Failed Objects.

    5. Click Run.

      Look for Files failed to backup to determine the failed files.

Solution

Follow the steps given below to resolve this issue:

  1. Make sure that the Use VSS options is enabled in the General tab of the Subclient Properties dialog box prior to performing system state backup.

  2. Make sure that \\computername\ClusterLegacyBackup share is accessible from the node that is being backed up.

  3. Make sure that the correct computer name and is given in the UNC path and that it resolves to the correct IP address.

  4. If the client being backed up is a virtual machine, disable system state backups for that machine.

  5. Make sure that the Cluster services are up and running.

    Note

    For cluster computers operating on Windows 2003, you can use only Legacy method to backup the cluster component.

Additional Information

System state backups are capable of backing up both the active and passive nodes in a cluster database. When a system state backup is initiated, the active owner generates a dump of the Cluster database and places it on a network share. This dump is then retrieved and backed up by the node that performs the backup.

Cluster Database backup failures are caused mainly due to an inability to access these shares during the backup. You will observe the following error in clbackup.log:

<--- STARTING BACKUP OF Cluster Database ---> 
 Cluster dump at [\\computername\ClusterLegacyBackup] and backup path [C:\Program Files\{Software Dir}\Commvault\iDataAgent\FileSystemAgent\2\123\clusterdb] 
 [::clusterBackup]: Name: NAME 
 ::clusterBackup() - BackupClusterDatabase call failed with following error code [51] 
 <--- ENDING BACKUP OF Cluster Database - [FAILED] --->

Error codes 52, 53 and 1214 are all connectivity errors.

System Protected Files on Windows 2008 Server
Symptom

System state backup performed on Windows 2008 servers may fail to backup System Protected Files.

This error is caused by Access Denied Errors during backup for Cryptographic Service. You may observe the following errors:

  • CommCell Console 

    Error Code 17:55:
 Failed to create a VSS snapshot and it is required in order to run a system state backup. Failing the scan.

  • clbackup.log 

    FsBackupTw::BackupComponentTemplate[System Protected Files]() - FsBackupTw::BackupComponentTemplate[System Protected Files]() - Backing up [System Protected Files]
 FsBackupTw::BackupComponentTemplate[System Protected Files]() -
 Entering [CSystemProtectedFilesWriterBackup::Backup]
 Entering [CDefaultWriterBackup::Backup]
 Entering [CDefaultWriterBackup::BackupWriter]
 Entering [CDefaultWriterBackup::GatherFlags]
 Entering [CDefaultWriterBackup::GatherWriterExcludes]
 Entering [CsVssBackupComponents::CsVssBackupComponents]
 Entering [CsVssBackupComponents::Delete]
 Entering [CsVssBackupComponents::GetWriter]
 Entering [CsVssBackupComponents::GetWriter]
 Call GetWriter(writerGUID, pWriter) [FAILED, throwing CV exception] - Code=0x80004005, Description=E_FAIL
 CDefaultWriterBackup::GatherWriterExcludes() - vss.GetExcludeFiles(writerGuid, lExcFiles) - [FAILED] 0x80004005
 Entering [CsVssBackupComponents::~CsVssBackupComponents]
 Entering [CsVssBackupComponents::Delete]
 CsVssBackupComponents::Delete() - Assigned set, so not freeing metadata
 CDefaultWriterBackup::BackupWriter() - GatherWriterExcludes(args, writerGuid) - [FAILED] 0x80004005
 FsBackupTw::BackupComponentTemplate[System Protected Files]() - FsBackupTw::BackupComponentTemplate[System Protected Files]() - Failed to backup System Protected Files
 FsBackupTw::BackupComponentTemplate[System Protected Files]() -
 FsBackupTw::DoSystemStateBackup() - Critical error detected in component [System Protected Files]. Failing system state backup
 PrintReport() - Component Method Status

  • filescan.log 

    Entering [CSystemProtectedFilesWriterBackup::Backup]
 Entering [CDefaultWriterBackup::Backup]
 Entering [CDefaultWriterBackup::BackupWriter]
 Entering [CDefaultWriterBackup::GatherFlags]
 Entering [CDefaultWriterBackup::GatherWriterExcludes]
 Entering [CsVssBackupComponents::CsVssBackupComponents]
 Entering [CsVssBackupComponents::Delete]
 Entering [CsVssBackupComponents::GetWriter]
 Entering [CsVssBackupComponents::GetWriter]
 Call GetWriter(writerGUID, pWriter) [FAILED, throwing CV exception] - Code=0x80004005, Description=E_FAIL
 CDefaultWriterBackup::GatherWriterExcludes() - vss.GetExcludeFiles(writerGuid, lExcFiles) - [FAILED] 0x80004005
 Entering [CsVssBackupComponents::~CsVssBackupComponents]
 Entering [CsVssBackupComponents::Delete]
 CsVssBackupComponents::Delete() - Assigned set, so not freeing metadata
 CDefaultWriterBackup::BackupWriter() - GatherWriterExcludes(args, writerGuid) - [FAILED] 0x80004005

  • Application or Event logs 

    Product: Windows Operating System 
 ID: 512 
 Source: Microsoft-Windows-CAPI2 
 Version: 6.0 
 Symbolic Name: MSG_SYSTEMWRITER_INIT_FAILURE 
 Message: The Cryptographic Services service failed to initialize the VSS backup "System Writer" object.%1.
    Product: Windows Operating System 
 ID: 513 
 Source: Microsoft-Windows-CAPI2 
 Version: 6.0 
 Symbolic Name: MSG_SYSTEMWRITER_ONIDENTITY_FAILURE 
 Message: Cryptographic Services failed while processing the OnIdentity() call in the System Writer Object.%1.
    Log Name: Application
 Source: Microsoft-Windows-Backup
 Event ID: 517
 Level: Error
 Description:
 Backup started at '...date/time..' failed with following error code '2155348226' (System writer is not found in the backup.). Please rerun backup once issue is resolved.
Solution

These errors appear more than once if they are caused by a permission issue or when any of the system writers is missing from the list of writers.

These writers are required to backup System Protected Files.

Run the following command and review the output.

vssadmin list writers > c:\vssadmin.txt

For more information, see Microsoft article Vssadmin.

The output on Windows 2008 server would look similar to the one given below.

Writer Instance Id: {1754f85c-8bb4-4928-8b54-9536c984298c}
 State: [1] Stable
 Last error: No error
 Writer name: 'Registry Writer'
 Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
 Writer Instance Id: {8c32dbe3-f944-4150-be3e-4bfc02b7295e}
 State: [1] Stable
 Last error: No error
 Writer name: 'COM+ REGDB Writer'
 Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
 Writer Instance Id: {cd07fee3-a9f6-4424-b42e-2dad49d8a311}
 State: [1] Stable
 Last error: No error
 Writer name: 'Shadow Copy Optimization Writer'
 Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
 Writer Instance Id: {ba399c41-f034-40c4-904d-9ae102c8bb1c}
 State: [1] Stable
 Last error: No error
 Writer name: 'WMI Writer'
 Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
 Writer Instance Id: {199b8675-695c-4c62-9a55-5230b31d69bc}
 State: [1] Stable
 Last error: No error
 Writer name: 'BITS Writer'
 Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
 Writer Instance Id: {6a8ac8d7-c598-4efd-913b-971148e89500}
 State: [1] Stable
 Last error: No error

For more information on system writers, see Microsoft KB article 2009272.

For Application or Event ID solutions, see Event ID 512 and Event ID 513.

For more information on missing system writers, see DPM 2007 Server 2008 System State Backup System Writer missing.

For more information on how VSS works, see How Volume Shadow Copy Works.

WFS0019: VSS errors

This is a collection of issues and resolutions of several VSS errors generated by Microsoft Windows operating system.

There are several VSS errors that generate the following errors:

  • CommCell Console

    Error Code 17:55
 Failed to create a VSS snapshot and it is required in order to run a system state backup. Failing the scan.

  • JobManager.log 

    Scheduler Set pending cause [Failed to create a VSS snapshot and it is required in order to run a system state backup. Failing the scan.]::Client [server name] Application [FileScan] Message Id [285212727] RCID [0] ReservationId [0]. Level [0] flags [0] id [0] overwrite [0] append [0] CustId[0].
 Scheduler Phase [Failed] message received from [server name] Module [FileScan] Token [37376:4:1] restartPhase [0]
 JobSvr Obj Phase [4-Scan] for Backup Job Failed. Backup will continue with phase [Scan].
Symptom 1

System state backup of an active partition that is not NTFS formatted can result in backup failure.

Resolution 1

See Backup Failures on Windows 2008, Windows 2008 R2 or Windows 7 for more information.

Symptom 2

The following errors appear when the writers are in a bad or missing state.

  • Writer errors 

    Writer name: 'System Writer'
 Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
 Writer Instance Id: {a9e9ad80-b17f-43fc-abde-693530722a5f}
 State: [9] Failed
 Last error: Timed out
    Writer name: 'WMI Writer'
 Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
 Writer Instance Id: {3ded1ea5-65c9-4e2b-bc0a-3064690a3eea}
 State: [9] Failed
 Last error: Timed out
    Writer name: 'Cluster Database'
 Writer Id: {41e12264-35d8-479b-8e5c-9b23d1dad37e}
 Writer Instance Id: {448f854c-ed09-439e-aeae-0e35e952ddab}
 State: [5] Waiting for completion
 Last error: No error

  • FileScan.log 

    CsSnapRequestor::GatherWriterStatus() - Gathering writer status After DoSnapshotSet... 
 CsSnapRequestor::GatherWriterStatus() - 10 writers responded
 STABLE status for writer 'Task Scheduler Writer'. code S_OK
 STABLE status for writer 'VSS Metadata Store Writer'. code S_OK
 STABLE status for writer 'Performance Counters Writer'. code S_OK
 FAILED_AT_FREEZE status for writer 'ASR Writer'. code VSS_E_WRITERERROR_TIMEOUT
 CsSnapRequestor::GatherWriterStatus() - Signaling bad state returned for writer [ASR Writer] engaged in backup.
 FAILED_AT_FREEZE status for writer 'System Writer'. code VSS_E_WRITERERROR_TIMEOUT
 CsSnapRequestor::GatherWriterStatus() - Signaling bad state returned for writer [System Writer] engaged in backup.
 FAILED_AT_FREEZE status for writer 'Shadow Copy Optimization Writer'. code VSS_E_WRITERERROR_TIMEOUT
 FAILED_AT_FREEZE status for writer 'COM+ REGDB Writer'. code VSS_E_WRITERERROR_TIMEOUT
 CsSnapRequestor::GatherWriterStatus() - Signaling bad state returned for writer [COM+ REGDB Writer] engaged in backup.
 WAITING_FOR_BACKUP_COMPLETE status for writer 'Registry Writer'. code S_OK
 WAITING_FOR_BACKUP_COMPLETE status for writer 'Cluster Database'. code S_OK
 FAILED_AT_FREEZE status for writer 'WMI Writer'. code VSS_E_WRITERERROR_TIMEOUT
 CsSnapRequestor::GatherWriterStatus() - Signaling bad state returned for writer [WMI Writer] engaged in backup.
 Call GatherWriterStatus("After DoSnapshotSet") [FAILED, throwing CV exception] - Code = 0x80004005, Description = E_FAIL
 Entering [CsSnapRequestor::DeleteFolderMetadata]
 CsSnapRequestor::DeleteFolderMetadata() - Not attempting to delete folder metadata
 Call DoShadowSet() [FAILED, throwing CV exception] -Code = 0x80004005, Description = E_FAIL
 Call CreateSnapshotSet() [FAILED, throwing CV exception] -Code = 0x80004005, Description = E_FAIL
 Entering [CsSnapRequestor::BackupComplete]
Resolution 2

Use writers.txt file to determine the writer causing the problem. For more information, see System Protected Files on Windows 2008 Server.

Symptom 3

This issue is mainly caused by a Diskshadow with a malformed path. For example, the following are registry keys available at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services where the file path is malformed.

  • Filelist.txt

    - File List: Path = c:/progra~2/plixer~1\cgi-bin, Filespec = eventlogd.exe 
 - File List: Path = archivemanagerremoteinstaller, Filespec = remoteinstallerservice.exe 
 - File List: Path = c:/program files (x86)/ibm_ds/ibmstoragemanagerprofiler server\bin, Filespec = tomcat5.exe 
 - File List: Path = c:\windows\\systemroot\system32\drivers, Filespec = lsi_sas.sys 
 - File List: Path = \systemroot\c:\windows\system32, Filespec = svchost.exe

  • FileScan.log 

    316 440 07/03 12:48:04 42346 CsSnapRequestor::AddVolumesToSnapshotSet() - Call m_vss->IsVolumeSupported(m_shadowProvider.GetProvider(), vol, &bSupported) [FAILED, throwing CV exception] - Code = 0x80042308, Description = VSS_E_OBJECT_NOT_FOUND
 316 440 07/03 12:48:04 42346 CsSnapRequestor::CreateSnapshotSet() - Call AddVolumesToSnapshotSet(ids) [FAILED, throwing CV exception] - Code = 0x80042308, Description = VSS_E_OBJECT_NOT_FOUND
Resolution 3

  1. Use shadow.txt to check for incorrect statements in the File Path section.

  2. Correct the malformed paths registry keys. Make sure to contact the Vendors owning the files as their installation has caused the malformed path.

For example:

HKEY_LOCAL_MACHINE\SYSTEM\ControlSet001\services
HKEY_LOCAL_MACHINE/SOFTWARE\Wow6432Node

Any path that has a / instead of \ does not conform to Windows LFS path standard. When the system writer identifies any malformed path, it cannot quiesce the file system for the VSS provider to take the snapshot.

Symptom 4

The following errors appear due to space issues with the Volume Shadow Service.

  • Server operating system Event Log File 

    TYPE] Error [TIME] 2011/09/23 09:03:04 [SOURCE] volsnap [COMPUTER] server name [DESCRIPTION] When preparing a new volume shadow copy for volume ?:, the shadow copy storage on volume ??: could not be located in non-critical space. Consider using a shadow copy storage volume that does not have any shadow copies.

  • FileScan.log 

    23924: [PID] [Thread] 09/23 09:03:05[JobID] CsVssAsync::WaitUntilDone() - Async status returned code = 0x8004231f, Description = Unknown status.
 23924: [PID] [Thread]09/23 09:03:05 [JobID] CsSnapRequestor::DoShadowSet() - Call asyncShadow.WaitUntilDone() [FAILED, throwing CV exception] - Code = 0x8004231f, Description = VSS_E_INSUFFICIENT_STORAGE
 23925: [PID] [Thread]09/23 09:03:05 [JobID] CsSnapRequestor::CreateSnapshotSet() - Call DoShadowSet() [FAILED, throwing CV exception] - Code = 0x8004231f, Description = VSS_E_INSUFFICIENT_STORAGE
 23926: [PID] [Thread]09/23 09:03:05 [JobID] CsSnapRequestor::Snap() - Call CreateSnapshotSet() [FAILED, throwing CV exception] - Code = 0x8004231f, Description = VSS_E_INSUFFICIENT_STORAGE
Resolution 4

Follow the steps given below to resolve this issue.

  1. Open My Computer on the client computer.

  2. Right-click on the <volume> and point to Properties.

  3. Click on Shadow Copies tab.

  4. Click Settings.

  5. In the Storage area, select Maximum Size as No Limit or the set the space limit required for the VSS copies. You can also move the holding section to another volume.

    vss_space_issues

For Command Line, see Microsoft article Vssadmin add shadowstorage.

Symptom 5

When multiple VSS Providers are active on the server, you will notice the following in FileScan.log.

Description = VSS_E_UNEXPECTED_PROVIDER_ERROR
 FindWorkThreadWrapper::w2kRecurse(4276): -Debug-: "C:\Users\addmtool\AppData\Local\History" is a junction point for "C:\Users\addmtool\AppData\Local\Microsoft\Windows\History".
 Call CreateSnapshotSet() [FAILED, throwing CV exception] - Code = 0x8004230f, Description = VSS_E_UNEXPECTED_PROVIDER_ERROR 
 Resolution 5
Cause

Run the following commands from the Command Prompt as an administrator for Windows 2008 and Windows 2008 R2 client computers.

vssadmin list writers > c:\writers.txt
vssadmin list providers > c:\providers.txt
vssadmin list volumes > c:\volumes.txt
diskshadow /L c:\shadow.txt
 list writers detailed
 exit

For Windows 2003 run the following commands:

vssadmin list writers > c:\writers.txt
Mountvol > C:\moutvol.txt
VSSADMIN List SHADOWS > C:\Shadows.txt
VSSADMIN List SHADOWSTORAGE > C:\ShadowStorage.txt
VSSADMIN list providers > C:\providers.txt

Verify if the commands run from the command prompt display results similar to the following:

  • Volumes.txt 

    Error: Unexpected failure: The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

  • Writers.txt 

    Error: A Volume Shadow Copy Service component encountered an unexpected error.
 Check the Application event log for more information.

  • Providers.txt

    Error: Unexpected failure: The service cannot be started, either because it is disabled or because it has no enabled devices associated with it.

  • DiskShadow.txt 

    DISKSHADOW> list writers detailed 
 COM call "(*vssObject)->InitializeForBackup" failed. 
 The last operation failed.
 - Returned HRESULT: 80042302
 - Error text: VSS_E_UNEXPECTED 
 DISKSHADOW> exit
 riters detailed
Resolution 5

These issues are caused mainly due to VSS Services not starting or running properly.

Follow the steps given below to resolve these issues:

  1. Make sure that the following services are not disabled.

    • Microsoft Software Shadow Copy Provider

    • Remote Procedure Call (RPC)

    • Volume Shadow Copy

  2. Check the Application and System event logs for VOLSNAP/VSS errors.

  3. Check the Providers installed using the following command: 

    VSSADMIN LIST PROVIDERS

  4. Check the status of writers using the following command: 

    VSSADMIN LIST WRITERS

    If you are still unable to resolve this issue, contact Microsoft.

    For more information, see Volume Shadow Copy Service, Volume Shadow Copy Service Operations for Windows 2008 R2, Volume Shadow Copy Service Operations for Windows 2008 R2.

Symptom 6

When multiple VSS Snap Providers are active on the server, you will notice the following in FileScan.log.

Entering [CsSnapRequestor::DeleteFolderMetadata]
 CsSnapRequestor::DeleteFolderMetadata() - Skipping delete of folder metadata.
 Call DoShadowSet() [FAILED, throwing CV exception] - Code = 0x80042306, Description = VSS_E_PROVIDER_VETO
 Call CreateSnapshotSet() [FAILED, throwing CV exception] - Code = 0x80042306, Description = VSS_E_PROVIDER_VETO

When you run the VSSADMIN PROVIDERS LIST from the command prompt, the output would be similar to the following:

vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
 (C) Copyright 2001-2005 Microsoft Corp. 
 Provider name: 'Microsoft Software Shadow Copy provider 1.0'
 Provider type: System
 Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
 Version: 1.0.0.7 
 Provider name: 'Data ONTAP VSS Hardware Provider'
 Provider type: Hardware
 Provider Id: {ddd3d232-a96f-4ac5-8f7b-250fd91fd102}
 Version: 6. 3. 0. 4601
Resolution 6

nUseVSSSoftwareProvider additional settings must be added on the Client computer which has multiple VSS Providers on it. If the client machine has more than one Agent installed on it, then this additional setting must be created multiple times under the corresponding agents.

For more information on this additional setting, see nUseVSSSoftwareProvider.

Additional information

You may refer to the following links for more information.

WFS0020: Folder or File Names That Are Specified as Wildcard Characters in [A-Z] Format Are Not Backed Up

Refer to Knowledge Base article WFS0020: Folder or File Names That Are Specified as Wildcard Characters in [A-Z] Format Are Not Backed Up.

UFS0002: Error registering job with the Communications Service

Symptom

  • Backup fails with these error messages under the main error code 19:857.

    19:210"Error registering job with the Communications Service."

  • Additional error messages seen from Events tab in the Backup Job Details window.

    17:2"File system scan Failed"

  • Filescan.log file on the client will show this error:

    Failed to translate "servername" to an IP address. Host ignored.
 Find::impersonateUser()() - No user id found from the registry [VMName = ... ]. Trying getting it from the CS
 ::GetSubclientDir() - The Subclient directory is [d:\Program Files\Commvault Systems\Galaxy\iDataAgent\JobResults\CV_JobResults\iDataAgent\FileSystemAgent]
 GetCommServerName() - CS name is [...]
 Overriding CommServer name [server name] with registry value [registry server name]
 JM Client CVBkpJobClient::init(): Initializing job object with token [...].
 CVJobCtrlClient::registerJob() CVD replied with a non-zero error code [1] for the job registration message. Error = [Failed to get host name for client [...] for job [...]]
 Init() - Error registering JOB with the Communications Service for JobID [...]
 Init() - registerJob failed. Returning failure
 Could not initialize job control.
 BKP CALLED COMPLETE (PHASE Status::FAIL), ... . Token [...]

  • JobManager.log file on the CommServe will show this error:

    Scheduler Set pending cause [Failed to get host name for client [...] for job [...]]::Client [...] Application [FileScan] Message Id [318767961] RCID [0] ReservationId [0]. Level [0] flags [0] id [0] overwrite [0] append [0] CustId[0].
 Scheduler Ignored pending cause [Insufficient information to start the job. Possibly mismatched version with CommServe or corrupted install.]::Client [...] Application [FileScan] Message Id [285212706] RCID [0] Reservation Id [0]. Level [0] flags [0] id [0] overwrite [0] append [0] CustId[0].
 Scheduler Phase [Failed] message received from [...] Module [FileScan] Token [...] restartPhase [0]
 AppManager JMAppMgr::getValidateArchiveIndexPhase() failed. Returning false
 JobSvr Obj Phase [4-Scan] for Backup Job Failed. Backup will continue with phase [Scan].

  • From the logs these additional error codes are part of the message:

In the Filescan.log file on the client

30:98 Failed to initialize Job Control.

Error message variations based on Agent:

7:28 Could not initialize job control object.
39:37 Could not initialize job object.

In the JobManager.log file on the CommServe

17:34 Insufficient information to start the job. Possibly mismatched version with CommServe or corrupted install.

Error message variations based on Agent/ MediaAgent/ or CommServe involved:

19:1033 Failed to get the host name.
 
19:104 Failed to get the target MediaAgent name for job [].
 
19:1102 Failed to get the MediaAgent name. Please verify that the MediaAgent version is compatible with the client version.
19:249 Failed to get the CommServe host name from the platform information.

  • For a restore job, you may receive this error in the clrestore.log file:

    23:84 Restore: Wrong parameters for the job. Possibly mismatched version with CommServe or corrupted install.

  • When testing connectivity to the client computer with issues make sure to test name resolution in the following communications paths:

    • CommServe to Client, and from Client to CommServe

    • CommServe to MediaAgent, and from MediaAgent to CommServe

    • Client to MediaAgent ,and from MediaAgent to Client

The error is caused by DNS name resolutions not functioning properly in all directions. DNS or equivalent Lookup and Reverse Lookup must be functioning.

Local Windows Server Resolver Cache on each computer in the environment must be checked and Hosts files must be reviewed.

Resolution

Check the configuration in the CommCell Console for the CommServe, MediaAgent and the Client with issues. Different phases of the backup or restore operations require reviewing different communications paths. Also, check the registry entries on the CommServe, MediaAgent and the Client.

The example below will cover the CommServe and the client computers.

  1. From the CommCell Browser, right click the CommServe and click Properties. Check the CommCell Host Name.

    kb1

  2. From the CommCell Browser, right click the Client computer and click Properties. Check the Host Name and CommServe Host Name.

    kb2

  3. The registry on the CommServe (CS) will have two locations that show the CS Host Name when it also has a Client (CL) installed.

    For the CommServe Name used by the Clients other then the CommServe Client the registry entry for sCSHOSTNAME is located under \HKLM\Software\Commvault Systems\Galaxy\Instance001\CommServe

    kb3

    For the CommServe with a Client (CL) File System Agent installed, the name is hosted at this registry location.

    Registry entry for wDefaultHostName is located under \HKLM\Software\Commvault Systems\Common

    kb4

  4. Now to get the information for the MediaAgent (MA) or the Client (CL), locate the MA or CL in the CommCell Browser window, right click, and then click Properties. On the General tab, the following will be displayed.

    Client used as example here:

    kb2

  5. Now once the MediaAgent (MA) and/ or Client (CL) CommCell information is confirmed, go to the MA and/or CL and review the registry entries for the Name.

    Client computer is used for example here.

    • To confirm the CL Host Name, check the registry on the CL for the entry wDefaultHostName located under \HKLM\Software\Commvault Systems\Common.

      kb5

    • Now confirm the CommServe Hostname on the CL in the registry for the entry sCSHOSTNAME located under \HKLM\Software\Commvault Systems\Galaxy\Instance001\CommServe

      kb6

    • For UNIX/ Linux based MA or CL, running the commvault status command will display the CommServe (CS) name as it appears in the /etc/CommvaultRegistry/Galaxy/Instance001/CommServe folder. The .properties file at this location contains the setting. 

      [root@lx5 ~]# commvault status

  6. Now that the CommServe Hostname and the Host Name are confirmed, first confirm DNS supports Lookup and Reverse Lookup for the names. If this is correct and communication works in all directions based on ping, ping -a and nslookup, retry the backup or restore operation.

    • If this does not resolve the issue then review the Name Management documentation to get the names matching in the environment.

    • If this does not resolve the issue then move to first using IP addresses between the CS, MA and CL instead of names.

    • If this fails to resolve the issue then use host file entries to populate the Resolver cache with the name translations.

      Enter in this format

      <IP Address> <Short Name> <FQDName>

Confirm the issue is resolved by running a backup or restore job and checking the Filescan.log, JobManager.log and clRestore.log files.

UFS0005: After an authentication failed error if a MediaAgent or a client cannot communicate with CommServe it will disable itself immediately

Symptom:

During Backup:

  • Error Code19:1327

    Attempt start error: [...]

  • Error Code 9:90

    Authentication failed for host [...]. Network password does not match.

  • Error Code 40:52

     
    Failed to contact CommServe Services from the MediaAgent[...] during the start of the Data Transfer operation. Please check the network connectivity from this MediaAgent to the CommServe and make sure services are running on the CommServe.

  • Error Code 22:30

    Could not establish connection to [...] on attempt [...].

  • Error Code 19:1131

    Waiting for the services on the client [...] to come online.

  • Error Code 19:1597

    Failed to start phase [...] on [...] due to network error [...]. Will attempt to restart. Please check if this product's services are running on the remote host.

  • Error Code 7:77

    Unable to run [...] on the client. [CVSession::authenticateClient]:Remote system [FQDNservername]. Failed authentication returned from server.

  • Error Code 7:102

    Unable to run [...] on the client.

  • Error Code 7:111

    Unable to run [...] on the client. [...]
    Error Code 9:36 [...]:Remote system [...]. Could not get the password from platform information or the database - authentication failed.
    Error Code 9:37 [...]:Remote system [...]. Could not find the client password in the database - authentication failed.
    Error Code 9:38 [...]:Remote system [...]. Failed authentication returned from server.
    Error Code 9:39 [...]:Remote system [...]. Failed authentication returned from client.
    Error Code 9:40 [...]:Remote system [...]. Unexpected message type received [...=...] - authentication failed.
    Error Code 9:41 [...]:Remote system [...]. Could not encrypt challenge string when connecting - authentication failed.
    Error Code 9:42 [...]:Remote system [...]. Password received from client did not match the database password for that client - authentication failed.
    Error Code 9:89 Authentication failed for host [...]. Could not find the client entry in database.
    Error Code 9:90 Authentication failed for host [...]. Network password does not match.
    Error Code 9:91 CVSession Authentication failed for host [...]. Password is not available on the host.

During Restore:

  • Error Code 23:98

    Seek failure happened

Additional Log Cuts:

  • From the JobManager.log file on the CommServe:

    6956 2764 07/23 16:00:21 53303 Servant [---- SCHEDULED BACKUP REQUEST ----], taskid [84] Clnt[...] AppType[Windows File System][33] BkpSet[defaultBackupSet] SubClnt[default] BkpLevel[Incremental][2]
 6956 27dc 07/23 16:00:24 53303 Scheduler Phase [4-Scan] could not start on [CLFQDNservername] due to Network Failure

  • From the CVD.log file on the Client:

    8000 1f64 07/25 13:14:33 ### Unable to resolve remote host_Disabled_CSFQDNServerName to any IP family: [1][11001][No such host is known.]
 8000 1f64 07/25 13:14:33 ### ** CVSsession::attach(ulPortArg):
 - RemoteHost=_Disabled_CSFQDNServerName. - RemoteProcess=. Port =65535,65535
 - getConnection() failed. Error=900004b.
 8000 1f64 07/25 13:14:33 ### ** CVSession::getClientPassword ():
 - RemoteHost=_Disabled_CSFQDNServerName.
 - RemoteProcess=JobMgr.exe.
 - commServerAttach (ePlatformTypeMember) failed. Error=900004b.
 8000 1f64 07/25 13:14:33 ### ** CVSession::replyAttach (PlatformType)
 - RemoteHost=CSFQDNServerName.
 - RemoteProcess=JobMgr.exe.
 - authenticateServer failed. Error=9000024.
Resolution:

If the nChatterFlag additional setting is set to 1, normal Communication and Connection testing will work properly but communication between MediaAgent or Client and the CommServe will continue to fail.

NOTE: The core issue is the MediaAgent or Client realized it was getting nowhere trying to talk (authenticate) to the CS and stops talking by setting the nChatterFlag and other additional settings. The cause of this type of behavior is outlined above but may also be caused by transient or extended CVD communication issues between elements of the CommCell and the CommServe.

To fix the issue, check the additional settings on the CommCell Components affected.

On Windows:

  1. Check the nChatterFlag additional setting.

    • From the CommCell Browser, navigate to Client Computers.

    • Right-click the <Client> or <MediaAgent>, and then click Properties.

    • Click Advanced and then click AdditionalSettings tab.

    • Click Add.

    • In the Name box, type nChatterFlag.

    • The Category and Type fields are populated automatically.

    • In the Value box, type 0.

      This enables communication to the CommServe.

    • Click OK.

  2. Check the sCSHOSTNAME additional setting.

    • From the CommCell Browser, navigate to Client Computers.

    • Right-click the <Client> or <MediaAgent>, and then click Properties.

    • Click Advanced and then click AdditionalSettings tab.

    • Click Add.

    • In the Name box, type sCSHOSTNAME.

    • The Category and Type fields are populated automatically.

    • In the Value box, if the current string value has _Disabled_ appended at the front of the string value, remove it to allow communication to the CommServe.

    • Click OK.

  3. If there is a firewall between the MediaAgent or Client and the CommServe, check the nDISABLE_CVFWD additional setting.

    • From the CommCell Browser, navigate to Client Computers.

    • Right-click the <Client> or <MediaAgent>, and then click Properties.

    • Click Advanced and then click AdditionalSettings tab.

    • Click Add.

    • In the Name box, type nDISABLE_CVFWD.

    • The Category and Type fields are populated automatically.

    • In the Value box, type 0.

      This enables controlled connection to the CommServe.

    • Click OK.

  4. Use the SCM utility to restart Commvault services.

On UNIX:

It is recommended to stop Commvault services on the MediaAgent(s) and/or Client(s) before modifying additional settings.

  1. Check the nChatterFlag additional setting.

    • From the CommCell Browser, navigate to Client Computers.

    • Right-click the <Client> or <MediaAgent>, and then click Properties.

    • Click Advanced and then click AdditionalSettings tab.

    • Click Add.

    • In the Name box, type nChatterFlag.

    • The Category and Type fields are populated automatically.

    • In the Value box, type 0.

      This enables communication to the CommServe.

    • Click OK.

  2. Check the sCSHOSTNAME additional setting.

    • From the CommCell Browser, navigate to Client Computers.

    • Right-click the <Client> or <MediaAgent>, and then click Properties.

    • Click Advanced and then click AdditionalSettings tab.

    • Click Add.

    • In the Name box, type sCSHOSTNAME.

    • The Category and Type fields are populated automatically.

    • In the Value box, if the current string value has _Disabled_ appended at the front of the string value, remove it to allow communication to the CommServe.

    • Click OK.

  3. If there is a firewall between the MediaAgent or Client and the CommServe, check the nDISABLE_CVFWD additional setting.

    • From the CommCell Browser, navigate to Client Computers.

    • Right-click the <Client> or <MediaAgent>, and then click Properties.

    • Click Advanced and then click AdditionalSettings tab.

    • Click Add.

    • In the Name box, type nDISABLE_CVFWD.

    • The Category and Type fields are populated automatically.

    • In the Value box, type 0.

      This enables controlled connection to the CommServe.

    • Click OK.

  4. Restart Commvault services.

  5. Test Backups and Restores to confirm data protection has been restored.

Special Case Scenario:

This resolution does not work if the pNET password on the MediaAgent or Client and the entry in the CommServe database do not match.

  1. Check the CVD.log on the MediaAgent or Client for these entries: 

    3756 d98 04/25 16:17:23 ### [ClientSessionWrapper::connectToCS] Error connecting to svc [AppManager network access] on CS. [150994982-[CVSession::authenticateClient]:Remote system [servernameFQDN]. Failed authentication returned from server.]
 3756 d98 04/25 16:17:23 ### ClientInterfaceNet::lookup() - [ClientSessionWrapper::connectToCS] Error connecting to svc [AppManager network access] on CS. [150994982-[CVSession::authenticateClient]:Remote system [servernameFQDN]. Failed authentication returned from server.]
 3756 d98 04/25 16:17:23 ### ::UpdateClientInfo() - Failed to do the client lookup for CLHostName
 3756 d98 04/25 16:17:23 ### ::UpdateClientInfo() - Session connect failed with error code 150994982 - [CVSession::authenticateClient]:Remote system [servernameFQDN]. Failed authentication returned from server., Generic error 150995034
 3756 d98 04/25 16:17:23 ### ::UpdateClientInfo() - Setting nChaterFlag to 1 to stop all communications
 3756 d98 04/25 16:17:23 ### ::UpdateClientInfo() - Setting nDISABLE_CVFWD to 1 to stop all communications
 3756 d98 04/25 16:17:23 ### ::UpdateClientInfo() - CS Host Name set to _Disabled_servernameFQDN to avoid any contact from this client to CS

  2. Check for a matching entry with same Date/Time stamp in the CVD.log file on the CommServe: 

    4532 92c 04/25 15:08:33 ### EvSANVolMgr::DiscoverFLRSupportedApps() - CVSession::attach() to [ipaddress*CLHostName*8400*8402] returned error 9000026
 4532 92c 04/25 15:08:33 ### ::sendResponse() - FAILED [Failed to connect to remote host [ipaddress*CLHostName*8400*8402].]
 3196 bc0 04/25 16:17:23 ### [CVD ] ** CVD_CVSESSION_ERROR: RemoteHost=ipaddress? Error replying to attach on socket: 900002a=[CVSession::authenticateServer]:Remote system [ipaddress]. Password received from client did not match the database password for that client - authentication failed.

WFS:0032 Backup Job Fails Due to Deleted or Skipped Files

Symptom

Backup job fails with the following error:

ERROR_FILE_NOT_FOUND
ERROR_PATH_NOT_FOUND
Cause

If certain files are found during scan phase and are missing during backup, then the backup of these files fail. Also, these files would be counted as failed objects.

Resolution

By default, the missing files are skipped during backup and are not reported as a failed item. However, if you have created the nDoNotSkipMissingFilesOnBackup additional setting on the client computer with a value set to 1, the missing files are reported as a failed item. Note that this key is applicable only for missing files and not for missing directories. Any missing directory found would be counted as an failed object.

WFS0033: Contents are not backed up although the backup completes successfully

Symptom

When you run a backup job on Windows computers, some content might not get backed up. However, the backup job completes successfully with no errors.

Cause

The subclient content might consist of entries that reside in the following location:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\BackupRestore\FilesNotToBackup.

Resolution

Any file located under the FilesNotToBackup folder are skipped during the backup job. However, the backup job completes successfully and does not report an error.

To include the skipped files in the backup, remove the required files from the following location:

HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\BackupRestore\FilesNotToBackup.

WFS0034: Windows operating system service failures

Symptom

Files backed up using VSS services and restored using Windows File System Agent might be corrupted.

Cause

Commvault leverages Microsoft Windows Volume Shadow Copy services (VSS) to generate snapshots of the data residing on the client computer. If this snapshot is corrupted, data might also get corrupted.

Resolution

Contact Microsoft to obtain hot fixes for this issue. For more information, see Microsoft KB article 2748349.

Additional information

For additional information, see Potential Sparse File Corruption on Windows operating system.

WFS0035: Backup failure of cluster computers

Symptom

After you perform a disaster recovery of cluster computers, backup of the virtual client that manages the cluster computers will fail.

Cause

The virtual client fails to connect to the cluster computers after you perform the System State recovery.

Solution

  1. Right-click the virtual client that manages the cluster computers and point to Properties.

  2. In the Client Computer Properties dialog box, click Advanced.

  3. In the Advanced Client Properties dialog box, click Cluster Group Configuration tab.

  4. Enable the Force Sync configuration on remote nodes checkbox.

  5. Click OK.

  6. Restart the services on both nodes of the cluster.

WFS0037: Backup Using Post-Backup Process Fails because of Using a Different MediaAgent

Symptom

A backup using post-process scripting fails because the Archive Index phase uses a MediaAgent in a GridStor setup that is different from the one used during the backup phase.

Cause

If the customer uses a post-backup script on a 10.0 system that was upgraded from 9.0, the MediaAgent used during the backup is released when the backup phase completes.

Solution

Create and/or enable the HoldReservationDuringPostBkp Additional Setting. This ensures that the MediaAgent is "sticky"; that is, it is retained for the Archive Index phase rather than being released at the end of the backup phase. Follow the procedure in Add or Modify an Additional Setting, using a Value of 1 and confirming that the Enable option is selected.

WFS0038: Unable to Configure the Scale-Out File Server

Refer to Knowledge Base article WFS0007: Unable to Configure the Scale-Out File Server.

Loading...