DB2 Agent Restore Operations Troubleshooting

Restore Failures

Out of place restore failures

The restore to a new database or another instance from the CommCell Console fails.

Do the following:

Make sure that database is not created on a different directory other than the default location.
The DBHEAP, UTIL_HEAP, and/or APP_CTL_HEAP_SZ configuration parameters for the (source) database that you want to restore should contain the default value.

Viewing Restore Logs

You can troubleshoot restores by checking the CLDb2Agent.log restore log located in <software installation path>\Log Files. You can also check details on the restore by issuing the db2ckrst DB2 command and then reviewing the output. The output for this command lists all of the backup images involved in the restore. The command syntax is as follows:

db2ckrst [-d <database name>] [-t <timestamp string>]

where:

<database name> is the name of the database and <timestamp string> is a numerical value representing the backup image involved in the restore.

Timestamps are displayed in the Restore Options | Restore Arguments dialog box in the CommCell Browser.

For example:

In this example, the timestamp for the database image included in the command appears twice in the output — in the first line and in the last line. This indicates that this (delta or incremental) backup image was the last backup image in the backup cycle and therefore was restored first and then last among all of the backup images in the cycle.

db2ckrst -d db_one -t 20030224125749

might generate the following output:

Suggested restore order of images using timestamp 20030224125749

for database db_one

============================================================

restore db db_one incremental taken at 20030224125749

restore db db_one incremental taken at 20030224124314

restore db db_one incremental taken at 20030224125211

restore db db_one incremental taken at 20030224125749

=============================================================

General

DB2 iDataAgent issue with backup or restore - List of Items for Effective Troubleshooting

When you encounter a DB2 iDataAgent issue with backup or restore, provide the following information to the Customer Support for troubleshooting the issues:

Failed backup/restore Job Type, backup or restore with JOB ID number – Initiated either from CommCell Console or DB2 Command line.
Complete set of logs from the CommServe, MediaAgents and the DB2 iDataAgent having an issue. Do not collect logs by Time Range or JOB ID options as this will not provide the information required to troubleshoot a DB2 Backup or Restore issue.
The exact OS platform for UNIX or Windows servers that the DB2 Application is running on with FixPack or Maintenance levels.

Log on to the DB2 server using the DB2 user account configured in the CommCell Console and perform the following commands:

db2level 
 (This command will provide the current DB2 application level)
 db2 list db directory 
 (This command will provide all of the DB2 databases currently configured)
 db2 get db cfg for database_name 
 (The database_name for each database listed in the above command. This will need to be run multiple times, once for each database_name listed)

In most cases, you may be required to send the DB2 applications db2diag.log. This log is not part of the Commvault Software components. It is the DB2 application database log file.

DB20011: Restore Fails with Permissions Issue

Issue:

The restore fails due to issues accessing the Commvault registry, log files and base directories.

Solution

Run the Database Readiness Check.

DB20013 Target Database Cannot Access Latest Log Chain During a Cross-Machine Restore

Issue:

When the DB2 log chain is being used during a cross-machine restore and the target database does not have access to the latest log chain, the log retrieval restore fails during the backup.

Solution:

Prior to running the restore:

Ensure the latest DB2 FixPack is installed on the target DB2 database.
In the CommCell Console, delete the target database instance, which also removes references to previous data.
Delete the database on the target server and recreate it.
In the CommCell Console, re-create the instance and the backup set for the target database.
Re-run the cross-machine restore to refresh the production database to the target database.

DB20014: A Backup or Restore Fails when Third Party Authentication is Enabled

Issue:

A backup or restore fails during the log retrieval phase when third party authentication is enabled. The following error is displayed.

Description: Rollforward failed with DB2 error [ ---- error report ---- app. message = rolling forward database line = 6484 file = ClDb2AgentLib.cpp SQLCODE = -1268 SQL1268N Rollforward recovery processing has stopped because of error "SQL2079" while retrieving log file "S0000163.LOG" for database "JDB27351" on database partition "0" and log stream "0". --- end error report --- ] on client [db2winmac1] for DB [JDB27351].

Resolution

This can happen when the third party authentication is not enabled t the database where the log files reside. You must enable this setting on the client, instance and database. For information on how to configure the additional setting, see Enabling CommCell Console Authentication of Third Party Command Line Jobs.

DB20015: A Table Restore Fails Because theDB2 Instance Port Number Is Incorrect

Issue:

A DB2 table restore fails because the DB2 instance port number or SVCNUM is not configured with the correct value. The error is similar to the following in the ClDb2Agent.log when the debug level is greater than 7:

ClDb2Agent::ClDb2Agent::ExecuteDB2Scripts() - 32: Received(334 bytes):SQL30081N A communication error has been detected. Communication protocol being used: "TCP/IP". Communication API being used: "SOCKETS". Location where the error was detected: "172.19.63.216". Communication function detecting the error: "connect". Protocol specific error code(s): "10061", "*"

Resolution

Verify that you set the DB2 communications protocol (DB2COMM) to TCPIP. On the command line type the following:

DB2SET DB2COMM=TCPIP

If you set any of the following configuration items, you need to check that they have the correct value. For information on how to find the DB2 Instance port number, see How do I Find Out the DB2 Instance Port Number.

The DB2 SVCENAME configuration parameter on the target instance.
The DB2 instance port number. This is the Database Server Port box on the Table Restore Options tab of the Table Restore Options dialog box. For information on how to configure this option, see Restoring DB2 Tables to a Different Database on a Different Instance on the Source Host.

DB20016: A Table Restore Fails Because the Target Client is Not Installed or Configured in the CommCell Console

Issue:

A DB2 table restore to a new destination fails because the DB2 target client is not installed or configured on the CommCell Console. The error is similar to the following in the ClDb2Agent.log when the debug level is greater than 7:

C[ClDb2AgentLib.cpp:12619] 0xFFFFFFFF:{ClDb2Agent::ExecuteDB2Scripts(12618)/ErrNo.-1.(Unknown error 18446744073709551615)-system(/db2/simpana/Base/Temp/db2TblScript89255_1409308715.sh) failed with error (Bad file descriptor::SQL1336N The remote host "dbserve36.commvault.com" was not found. SQLSTATE=08001

Resolution

Install and configure the DB2 target client on the CommCell Console.

DB20017: A Table Restore Fails Because the Target Database is Not Configured in the CommCell Console

Issue:

A DB2 table restore to a new database fails because the DB2 target database is not configured on the CommCell Console. The error is similar to the following in the ClDb2Agent.log when the debug level is greater than 7:

C[ClDb2AgentLib.cpp:12716] 0xFFFFFFFF:{ClDb2Agent::ExecuteDB2Scripts(12715)/ErrNo.-1.(Unknown error 18446744073709551615)-system(/opt/simpana/Base/Temp/db2TblScript207583_1412848543.sh) failed with error (Bad file descriptor::SQL1013N The database alias name or database name "TTTT" could not be found. SQLSTATE=42705 SQL1024N A database connection does not exist. SQLSTATE=08003 SQL0843N The server name does not specify an existing connection. SQLSTATE=08003

Resolution

Configure the DB2 target database on the CommCell Console.

DB20018: A Restore from a Tape Library Hangs

Issue

A restore from a tape library hangs with the following error:

The medium is already reserved by some other job[s]

Resolution

This is due to default optimization that the software uses to enhance performance. To fix this issue, disable optimization. For more information, see Disabling Optimization for Database Restores.

DB20019: A Table Level Restore Shows Pending

Issue

A table level restore shows the Job Pending Reason (JPR) as:

Description: DB2 Restore [m_backupTimestamp.size()=0..]

Resolution

The reason that this happens is that the latest backup image was not from a subclient that has the Enable Table View option selected (Subclient Properties dialog box, Advanced Options tab, Enable Table Browse check box).

You can use the browse the job history for a table level backup or restore to a point-in-time that uses the table level backup.

To avoid this issue, create one subclient with the options hat are specified in this procedure for a backup set. This ensures that the backup image will be the right type for a table level restore.

DB20022: A Point-in-Time Restore Fails

Issue

A DB2 point-in-time restore fails because the target database has errors.

Resolution

We recommend that you drop the database before the restore by using the DB2 command "db2 drop database <db_name>".

DB20023: A DB2 Backup or Restore Fails when the Device is in CIO Mode

Issue

A DB2 backup or restore fails when the device is in CIO mode.

Resolution

Stop the DB2 instances.
Unmount, and then remount the deice by using the -o cio option.

Example
```
mount -v jfs2 -o cio /dev/Q03-db2data1 /db2/Q03/sapdata1
```

DB20024: DB2 Command Line Restore Fails with SQL2538N Error

Issue

This issue applies to DB2 10.5.8 and higher.

When you run a DB2 command line restore, you might get a "SQL2538N Unexpected End of File of Backup Image reached on media" error.

Resolution

This error occurs because during the DB2 backup, DB2 uses an extra session for the logs. This causes a session mismatch error when you restore the backup.

You must use an extra session when you perform the restore.

Example

In the following command, the backup image that you want to restore is the same name as the target database. This restore operation causes the software to overwrite the current database with the backup version. The restore must use one more session to avoid the error.

Original command

db2 restore db sample load c:\simpana\base\db2sbt.dll open 2 sessions taken at 20170829171737

Updated command

db2 restore db sample load c:\simpana\base\db2sbt.dll open 3 sessions taken at 20170829171737

Vendor Code Troubleshooting

Return Code 23

Issue:

The requested backup image or log has been successfully restored.

Solution

There is no action required.

Return Code 25

Issue:

There was an I/O error.

Solution

Check permissions on the DB2 archive paths.
Verify the disk space on the host that the data is being restored to.

Return Code 30

Issue:

There was a critical error.

11403590 304 05/12 18:03:18 1048366 Db2Check::SqlInfoPrint() - 0:
 ---- error report ----
 app. message = backing up the database
 line = 4804
 file = ClDb2AgentLib.cpp
 SQLCODE = -2079
 SQL2079N An error was reported by the shared library
 "/opt/commvault/Base64/libDb2Sbt.so". Return code: "30".
 --- end error report ---
 11403590 304 05/12 18:03:18 1048366 ClDb2Agent::DataBackup() - 1: SetPendingCauseAndEvent:DB2_SBTCRITICAL_ERROR

Solution

Contact support with the following information ready to send.
- The database configurations,
- the db2dump directory,
- all crash dumps