DB2 Agent Backup Operations Troubleshooting

Backup

DB20006: Log only backup fails after a DB2 upgrade from 32-bit to 64-bit

Issue

Log Only Backups fail after a DB2 upgrade from 32-bit to 64-bit.

Symptom

Log Only Backups fail after a DB2 upgrade from 32-bit to 64-bit.

Resolution

The sqllib/lib library is not linked to the correct 64-bit library. Link the sqllib/lib library to a correct lib64 for 64bit DB2 .

DB20007: Log backup fails

Issue

If you create the directories, subdirectories or files under Commvault Log Archive/Retrieve/Audit Error paths, the log backups may fail.

Symptom

Log Backups fail.

In the example, the "db2inst1/DB5290/NODE0000/C0000000//lost+found" directory exists under "/home/db2inst1/db2Archive" path. Hence, the log backups failed. The following error message is displayed in ClDb2Agent.log:

11443 40476940 11/12 16:20:52 192887 Db2UserExit::OpenLogFile() - 1: fopen(/home/db2inst1/db2Archive/db2inst1/DB5290/NODE0000/C0000000//lost+found) 11443 40476940 11/12 16:20:52 192887 Db2Util::Db2StringToULong() - 0: ERROR: db2str.size()=10 < i
 11443 40476940 11/12 16:20:52 192887 Db2UserExit::PopulateLogFileHeader() - 0: Util.Db2StringToULong(lost+found) failed.
 11443 40476940 11/12 16:20:52 192887 Db2UserExit::BackupLogsFromPath() - 0: PopulateLogFileHeader() failed.
 11443 40476940 11/12 16:20:52 192887 Db2UserExit::BackupFromAD() - 0: BackupLogsFromPath failed.
 11443 40476940 11/12 16:20:52 192887 Db2UserExit::Backup() - 0: BackupFromAD() failed: .
 11443 40476940 11/12 16:20:52 192887 ClDb2Agent::LogsBackup() - 0: Before delete ue...: ue->Backup() failed: errorHelpString=<>.
 11443 40476940 11/12 16:20:52 192887 ClDb2Agent::LogsBackup() - 1: SetPendingCauseAndEvent:DB2_LOGBACKUP_FAILED
 11443 40476940 11/12 16:20:52 192887 Db2JobObject::Db2JobObject::SetPendingCauseAndEvent() - 1: Setting the pending cause successful.

Resolution

If you have created any directories, subdirectories or files under Commvault Log Archive/Retrieve/Audit Error paths, move them to another location.

DB20008: Backup jobs fail with one or more errors

Issue

The backup job fails.

Symptom

Backup jobs from DB2 iDataAgent will be displayed as "Completed w/ one or more errors".

Resolution

This happens in the following cases:

Job is killed after creating some valid data archive files. For example: after running a database backup, if job is killed during the logs phase.
When an online/offline/command line backup job completes the data phase but fails on the log phase.
When a LOGS only backup job fails or is killed after you backup some logs.

DB20009: Database is in the BACKUP_PENDING state

Issue

The database is left in a BACKUP_PENDING state

Resolution

If you have modified he LOGARCHMETH1 parameter without performing an offline full backup after, the database is left in the BACKUP_PENDING state.

Perform an offline full backup.

DB20010: A DB2 Backup Fails with Error Code 19:1334

Issue:

The DB2 backup fails with the following error.

ERROR CODE [19:1334]: DB2 Restore [Error: wrong message=282 from PL.] 
Source: noandvdb03a, Process: DB2SBT

Solution:

Restart the DB2 services.

DB20012: Target Database Cannot Access Latest Log Chain During a Cross-Machine Restore

Issue:

When the DB2 log chain is being used during a cross-machine restore and the target database does not have access to the latest log chain, the log retrieval restore fails during the backup.

Solution:

Prior to running the restore:

Ensure the latest DB2 FixPack is installed on the target DB2 database.
In the CommCell Console, delete the target database instance, which also removes references to previous data.
Delete the database on the target server and recreate it.
In the CommCell Console, re-create the instance and the backup set for the target database.
Re-run the cross-machine restore to refresh the production database to the target database.

DB20020: A DB2 Log Backup on Windows or AIX Goes to a Pending state

Issue

The DB2 backup fails with the following error.

3276922 1   08/24 18:02:54 ### Db2BackupRestoreUE::RecvArchFileListByLogNumber() - 0: SetLastErr: m_archive->listDB2LogsByLogNumber() failed retcode=1

Solution

For AIX, deactivate and then activate the database.
For Windows, restart the DB2 services

Note

Make sure to back up the archive logs before you perform a restore operation with the log rollforward option.

DB20021: The Backup Shows an Error Even When It Succeeds

Issue

The DB2 backup shows the following error when the job fails or succeeds.

31916264 1   08/02 14:29:09 4702363 cvfw_decode_cmd() - ERROR: Failed to decode incoming command: missing end signature. 31916264 1   08/02 14:29:09 4702363 connect() - ERROR: Did not get expected CLNT_FAILED or CLNT_SUCCESS commands from daemon.

Resolution

Restart the DB2 instance.

DB20016: The connection to a DB2 Database Does Not Work and the CommCell Console Backup Fails

Issue

The connection to the DB2 database does not work, and the CommCell Console backup fails with the following error:

An error occurred while accessing the shared library "C:\PROGRAM FILES\COMMVAULT\CONTENTSTORE\BASE\DB2SBt.dl

Resolution

Verify the following configuration items:

The env PATH variable is set
The DB2 LOGARCHMETH1 parameter is set to simpana\base\db2sbt.dll

The db2vars.env file is located under the db2_home\sqllib\cfg folder

If the file exists, then you must set the Commvault base directory as part of the path.

Example

PATH=E:\simpana\Base;C:\ProgramData\Oracle\Java\javapath;%SystemRoot%\system32;%SystemRoot%;%SystemRoot%\System32\Wbem;%SYSTEMROOT% \System32\WindowsPowerShell\v1.0\;C:\Program Files\ibm\gsk8\lib64;C:\Program Files (x86)\ibm\gsk8\lib;D:\DB2V10.5\IBM\BIN;D:\DB2V10.5\IBM \FUNCTION;D:\DB2V10.5\IBM\SAMPLES\REPL INCLUDE=D:\DB2V10.5\IBM\INCLUDE;D:\DB2V10.5\IBM\LIB LIB=;D:\DB2V10.5\IBM\LIB CLASSPATH=.;D:\DB2V10.5\IBM\java\db2java.zip;D:\DB2V10.5\IBM\java\db2jcc.jar;D:\DB2V10.5\IBM\java\sqlj.zip;D:\DB2V10.5\IBM\java \db2jcc_license_cu.jar;D:\DB2V10.5\IBM\bin;D:\DB2V10.5\IBM\java\common.jar 
 LOCPATH=

Vendor Code Troubleshooting

Reason Code 1: Troubleshooting backup failures after a new DB2 Agent install

Issue:

The backup may fail due to insufficient permissions for accessing the Commvault registry, log files and base directories.
The semaphore, file limits are not set properly.

The Reason Code 1 is displayed with the following

[IBM][CLI Driver] SQL2071N An error occurred while accessing the shared library "C:\Program Files\CommVault\ContentStore\Base\Db2Sbt.dll". Reason code: "1".

Solution

Use the cvpkgchg utility to set correct group, access permissions for the Commvault installation. For more information, see Changing UNIX Groups and Permissions on UNIX Clients.
Set the semaphore, file limits as per the install prerequisites. See Configuring the Kernel Parameters for more information.
On Windows check the PATH environment variable has the Commvault path.

Return Code 6

Issue:

The requested backup image or log file cannot be found.

Solution

Verify the backup was taken.
Verify the backup has not been pruned.

Return Code 11

Issue:

The initialization failed.

Example

dbserve36:/home/db2/V9.7B64/db2inst5>db2 backup db testdb load /opt/commvault/Base/libDb2Sbt.so 
SQL2062N An error occurred while accessing media "/opt/commvault/Base/libDb2Sbt.so". Reason code: "11".

Solution

Verify the vendor parameters are specified correctly.
Verify the connectivity between the CommCell, CommServe computer and MediaAgent was not interrupted.
Verify the client services are running.
Verify the Commvault group is set to the db2admin group.
Execute database discovery to make sure the database has been discovered. For more information, see Enabling Automatic DB2 Database Discovery or Manually Discovering DB2 Databases.

Return Code 18

Issue:

An error occurred during the data transfer.

Example

dbserve36:/home/db2/V9.7B64/db2inst5>db2 backup db testdb load /opt/commvault/Base/libDb2Sbt.so

Solution

Verify the connectivity between the CommCell, CommServe and MediaAgent was not interrupted.
Verify resources are available on the MediaAgent.

Return Code 30

Issue:

There was a critical error.

11403590 304 05/12 18:03:18 1048366 Db2Check::SqlInfoPrint() - 0:
 ---- error report ----
 app. message = backing up the database
 line = 4804
 file = ClDb2AgentLib.cpp
 SQLCODE = -2079
 SQL2079N An error was reported by the shared library
 "/opt/commvault/Base64/libDb2Sbt.so". Return code: "30".
 --- end error report ---
 11403590 304 05/12 18:03:18 1048366 ClDb2Agent::DataBackup() - 1: SetPendingCauseAndEvent:DB2_SBTCRITICAL_ERROR

Solution

Contact support with the following information ready to send.
- The database configurations,
- the db2dump directory,
- all crash dumps

Effective Troubleshooting

When you encounter a DB2 iDataAgent issue with backup or restore, provide the following information to the Customer Support for troubleshooting the issues:

Failed backup/restore Job Type, backup or restore with JOB ID number – Initiated either from CommCell Console or DB2 Command line.
Complete set of logs from the CommServe, MediaAgents and the DB2 iDataAgent having an issue. Do not collect logs by Time Range or JOB ID options as this will not provide the information required to troubleshoot a DB2 Backup or Restore issue.
The exact OS platform for UNIX or Windows servers that the DB2 Application is running on with FixPack or Maintenance levels.

Log on to the DB2 server using the DB2 user account configured in the CommCell Console and perform the following commands:

db2level

(This command will provide the current DB2 application level)

db2 list db directory

(This command will provide all of the DB2 databases currently configured)

db2 get db cfg for database_name

(The database_name for each database listed in the above command. This will need to be run multiple times, once for each database_name listed

In most cases, you may be required to send the DB2 applications db2diag.log. This log is not part of the Commvault Software components. It is the DB2 application database log file.