> Expert > Backup and Restore Agents > Databases > Oracle > Oracle Backup Administrator Guide > Oracle Agent Troubleshooting

Troubleshooting Restores - Oracle Agent

Updated

This section provides information on troubleshooting restores. For information about viewing RMAN logs, see Viewing RMAN Log Files.

ORCL0022: Point-in-time browse failures

Issue

When encryption is enabled for the client, a point in time table browse operation fails with the following error message: 

 
Pass-phrase protection is on for client [80], but pass-phase was not specifiedr.

Resolution

Make sure that the pass phrase is exported to the MediaAgent when encryption is enabled for the client.

  1. From the CommCell Browser, right-click the client and select Properties.

  2. Click the Encryption tab.

  3. Click Via Pass-Phrase.

  4. Click Export.

  5. In the Destination Computer box, select the MediaAgent.

  6. In the Pass-Phrase box, type the pass-phrase used for encryption.

  7. In the Re-enter Pass-Phrase box, re-type the pass-phrase to confirm.

  8. Click Export.

  9. Click OK.

ORCL0023: Table restore failures

Issue

The table restores may fail with the following error message:

RMAN-05502: the target database must be mounted when issuing a DUPLICATE command.

Resolution

For Windows, make sure that the Oracle Services are running as Local System.

Use connect string instead of ‘/’ when you configure an instance. For example, sys/pwd12@orcl4.

ORCL0024: Database restore failures

Issue

A database restore fails when the Redirect, Rename and Recover options were selected at the same time.

Resolution

After performing an Oracle restore operation from the CommCell Console click the Refresh button on the Subclient Properties (Content) tab or run a backup after the restore operation has completed before proceeding with another restore. This is necessary to ensure that the CommCell Console recognizes the changes that were made to the Oracle database and control file, so that it reflects the current structure of the database to be restored, otherwise the restore will fail.

ORCL0025: Unable to create duplicate database

Issue

A duplicate database restore fails with error PLS-00553: character set name is not recognized

Resolution

Make sure that the character sets are the same between the location from where you are running RMAN, and the location of the target database. As this is an Oracle related issue, contact Oracle support for more information.

When you create a duplicate database use a proper connect string instead of ‘/’ for one of the instances. For example, sys/pwd12@orcl4. This is required when you perform a duplicate database restore operation since we need to connect two instances in the same RMAN operation.

ORCL0026: Unable to restart a duplicate database restore job

Issue

The restore job is failing with the following error:

RMAN-00571: ===========================================================
 RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
 RMAN-00571: ===========================================================
 RMAN-03002: failure of Duplicate Db command at XX/XX/XXXX 16:28:20
 RMAN-05500: the auxiliary database must be not mounted when issuing a DUPLICATE command
 RMAN>

Resolution

The Auxiliary database is remaining in mount state when a suspended restore job is resumed.

This issue may occur due to the wrong syntax value (extra Slash (/) value) after Oracle home. You will receive a different status when connected to a database instance, so the duplicate database restore is failing.

Remove the (extra Slash (/) value) after Oracle home in the syntax under instance properties.

For example:

If the Oracle Home was set to the following under instance properties.

/u01/app/oracle/product/11.2.0/dbhome/

You must update the syntax as following:

/u01/app/oracle/product/11.2.0/dbhome

ORCL0027: Job fails due to the sbtio.log size

Issue

Sometimes, jobs fail due to the increase in the size of sbtio.log file in the $UDUMP directory.

Resolution

Set the size limit for the sbtio.log file using the sMAXORASBTIOLOGFILESIZE additional setting. Once the specified size limit is reached, the sbtio.log file gets pruned automatically.

ORCL0028: Table restore intermittent failures

Issue

The table level restore operation may fail intermittently due to an error in the Oracle's DataPump utility and the following error message is displayed:

UDE-00008: operation generated ORACLE error 31623

ORA-31623: a job is not attached to this session via the specified handle

Resolution:

Set the sNODATAPUMPEXPORT additional setting to Y on the client and re-submit the job.

ORCL0033: Restore Error on UNIX when Oracle uses large pages

Issue

When restoring an Oracle database on UNIX clients, if the Switch database mode for restore option is selected to put the database in the correct mode during the restore, the database may not restart after switching the database mode. Also, the restore operation may fail with the following error message.

RMAN Script execution failed with error [RMAN-04014: startup failed: ORA-27137: unable to allocate large pages to create a shared memory segment]. Please check the Logs for more details.

Resolution

This issue occurs if the Oracle user has a higher ulimit configuration than the root user. To resolve this issue, apply the ulimit value of Oracle user for the restore using the following steps:

  1. From the CommCell Browser, navigate to Client Computers.

  2. Right-click the <Client>, and then click Properties.

  3. Click Advanced.

  4. Click the Additional Settings tab.

  5. Click Add.

  6. In the Additional Settings dialog box:

    • In the Name box, type OracleUser.

    • In the Category box, select or type OracleAgent from the list.

    • In the Type box, select String.

    • In the Value box, type the Oracle user name (for example, oracle).

  7. Click OK.

  8. Restart the Commvault Services on the client.

ORCL0050: A PDB Clone Operation Fails When the User Has the SYSBACKUP Privilege

Issue

When you perform a PDB clone operation with a user that does not have the SYSDBA privilege (for example, the user has the SYSBACKUP privilege), the backup fails when it runs the unplug script, but the job status does not reflect the failure.

Resolution

Repeat the clone operation after you grant the user that performs the operation the SYSDBA privilege. For more information, see Cloning an Oracle 12c Pluggable Database (PDB).

ORCL0051: Dropped PDB Restore Fails with RMAN-06813 Error

Issue

A restore of a dropped PDB fails with a RMAN-06813: could not translate error.

Resolution

Use the Oracle PDB clone feature to restore the PDB. For more information, see Cloning an Oracle 12c Pluggable Database (PDB).

Note: Select the Time Range option, and set the time to the most recent backup time when the PDB existed.

ORCL0047: Error 72:151 Returned for a PDB Clone Operation

Issue

An PDB clone operation returns the following error:

Error Code: [72:151] Description: Error :[Oracle Instance target connect string cannot be / while performing PDB Clone, when there is no RMAN catalog configured on target. .]

Resolution

If you do not have a catalog, then you cannot use "/" in the target catalog connect string. Change the Catalog Connect String (Restore Options dialog box, General tab, Catalog Connect String boxes).

ORCL0048: A Restore from a Backup That Has RMAN Backup Optimization Set Fails

Issue

If the RMAN BACKUP OPTIMIZATION parameter is set to ON, then RMAN does not back up the datafiles that have not changed from the previous backup. This behavior might cause restores to fail when previous backup jobs are aged.

Resolution

Set the RMAN BACKUP OPTIMIZATION parameter to OFF. For more information, go to the Oracle documentation website, Configuring Backup Optimization.

ORCL0030: Command line backup fails for large backups

Issue

Sometimes, the third party command line jobs may hang when you perform large restores.

Resolution

Sometimes, the third party command line jobs may hang when you perform large backups and restores.
This happens because ClOraControlAgent updates the job manager for every 100MB of data transfer and this causes the thread failure for large backups/ restores after transferring some of the data. The following exception is seen. clOraControlAgent.log: 

5710030 304 02/22 03:47:23 608119 OraAgentBase::NotifyCommServeJobContinue() - m_jobObject->setUnCompBytesToAdd(105119744) ...
 5710030 304 02/22 03:47:24 608119 CvThread::start_func() - Unhandled exception.
 5710030 405 02/22 03:47:37 608119 ClOraControlAgent::OnClientTimeout() - Got timed out while waiting for msg from client 0

Set sBYTESDIFFMBS additional setting <value> in MBs in OracleAgent/.properties. This will update the job manager at every <value> in MBs specified in the key.

ORCL0031: Unable to create a standby database

Issue

The standby database fails with the following error message:

temporary file TEMP01.DBF conflicts with file used by target database

Resolution

Set the Standby Role Initialization parameter, DB_FILE_NAME_CONVERT to add all the temp datafiles from the primary database location to the standby database location, as follows:

DB_FILE_NAME_CONVERT='<primary_database_temp_datafile_old_location>','

<standby_database_temp_datafile_new_location>

ORCL0032: The block size in the backup and restore are different

Issue

The Block Size during the backup and restore are different. The following error is shown.

Error Code: [19:1336]

Description: Oracle Restore [Job[xxxxx] thread[xxxxxxxx]: Did not read correct number of bytes, expected: xxxxxx, read: yyyyyy.]

Resolution

Use the same block size that you have used during the backup. For example, if you have used 128 KB as the block size during the backup, make sure to use the same 128 KB as the block size during the restore of that backup. An increase in the block size during the restore may cause the restores to fail.

ORCL0040: Validation Error When Running RMAN Scripts

Issue:

An error message containing "Provide Valid Token" is returned when an RMAN script runs.

Solution

A valid token file was not included in the request.

  1. Run the qlogin command with the token file option (-f) to obtain a token file.

  2. Use the CvQcmdTokenFile parameter with the token file that the qlogin command generates.

    For information on required and optional SBT parameters, see SBT Parameters.

ORCL0041: User Error When Running RMAN Scripts

Issue:

An error message containing "Provide competent user" is returned when an RMAN script runs.

Solution

The user does not have the correct permissions in the CommCell Console to run the backup job.

ORCL0052: An Out-of-Place RMAN Control File Restore Fails

Issue

If you perform an RMAN out-of-place control file restore that sets the database id and the database is in the OPEN mode, then the restore fails with an error that is similar to the following error:

RMAN> executing command: SET CONTROLFILE AUTOBACKUP FORMAT 
 RMAN> 
 executing command: SET DBID RMAN-00571: 
 =========================================================== 
 RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS 
 =============== 
 RMAN-00571: ===========================================================
 RMAN-03002: failure of set command at 02/03/2017 18:25:07
 RMAN-06188: cannot use command when connected to a mounted target database

Solution

In order to resolve the issues, you must customize the RMAN script of a control file restore.

  1. Perform a control file restore and follow the step that customizes the RMAN script. For more information, see Restoring Control and Server Parameter Files for Oracle.

  2. Customize the RMAN script by making the following change.

    Example

    The original script has the SET DBID line.

     
    SET DBID 1234567890; 
run { 
allocate channel ch1 type 'sbt_tape' PARMS="SBT_LIBRARY=/opt/commvault/Base/libobk.so,
BLKSIZE=1048576 ENV=
(CV_mmsApiVsn=2,CV_channelPar=ch1,ThreadCommandLine=
-cn test_client -cs CommServe_test)"
TRACE 2; send " -jm 131082 -a 2:804 -cl 27 -ins 116 -at 22 -j 315558 -jt
315558:2:1 -rcp 0 -ms 1 -p 1"; 
restore controlfile to '/mydata/controlfile_1.ctl' from autobackup ;
}
exit;

    The updated script does not have the line.

    run { 
allocate channel ch1 type 'sbt_tape' PARMS="SBT_LIBRARY=/opt/commvault/Base/libobk.so,
BLKSIZE=1048576 ENV=
(CV_mmsApiVsn=2,CV_channelPar=ch1,ThreadCommandLine=
-cn test_client -cs CommServe_test)"
TRACE 2; send " -jm 131082 -a 2:804 -cl 27 -ins 116 -at 22 -j 315558 -jt
315558:2:1 -rcp 0 -ms 1 -p 1"; 
restore controlfile to '/mydata/controlfile_1.ctl' from autobackup ;
}
exit;

ORCL0035: Restore Fails with Permissions Issue

Issue:

The restore fails due to issues accessing the Commvault registry, log files and base directories.

The RMAN restore fails because it cannot load the CommVault SBT Media Management library.

Solution

Run the Database Readiness Check.

ORCL0037: Multiple Jobs for Oracle Third Party Command Line Operations

Issue:

For Oracle 12c, when performing Oracle multiple streams for third party command line operations, multiple jobs may be kicked off.

Solution

Add the user to Local security policy

  1. From Local Security Policy, navigate to User Right Assignment.

  2. Right-click Act as part of the operating system and then select Properties.

  3. Click on Add User or Group and then click OK.

  4. Right-click Create a token object and then select Properties.

  5. Click on Add User or Group and then click OK.

  6. Right-click Replace a process level token and then select Properties.

  7. Click on Add User or Group and then click OK.

ORCL0038: Cross-SID Restore Fails with Oracle Version 11.2.0.1 or Lower

Issue

If you have an Oracle version that is 11.2.0.1 or lower, then an Oracle cross-SID restore might fail with the following error:

Error Code: [19:1336]
Description: Oracle Restore [Failed to execute the create controlfile script. Please run this script [shutdown immediate; startup nomount; @'/opt/commvault/Base/Temp//CreateControlFileScript_1522047061.sql'; exit; ] manually again from sqlplus, open the database with resetlogs and change the id using nid.]
Source: client3, Process: ClOraAgent

Resolution

Oracle versions 11.2.0.1 or lower does not creat a valid SQL entry when it creates the control file. For more information, see the Oracle support article https://support.oracle.com/epmos/faces/DocumentDisplay?_afrLoop=174110220812530&id=9009028.8&_afrWindowMode=0&_adf.ctrl-state=y58zg9vqu_53.

In order to fix the issue, upgrade to the latest Oracle version, and then retry the operation.

Optimizing memory allocation for table restores

When restoring large tables, the restore operation may fail if there is insufficient memory allocation for creating the auxiliary instance.

Use the following steps to optimize the memory allocation for the auxiliary instance:

Allocating Memory for Auxiliary Instance

By default, 16MB pool size is allocated for the auxiliary instance. Use the following steps to increate this size limit:

  1. From the CommCell Browser, navigate to Client Computers.

  2. Right-click the <Client> and then click Properties.

  3. Click Advanced.

  4. Click the Additional Settings tab.

  5. Click Add.

  6. On the Add Additional Settings dialog:

  7. In the Name box, type sLARGEPOOLSIZE.

  8. In the Category box, select or type OracleAgent from the list.

  9. In the Type box, select String.

  10. In the Value box, type the memory for the auxiliary instance (for example, 32M).

  11. Click OK.

Allocating Memory for Oracle Streams

By default, the system allocates 48 MB for the Oracle streams. You can modify this value using the following steps:

  1. From the CommCell Browser, navigate to Client Computers.

  2. Right-click the <Client> and then click Properties.

  3. Click Advanced.

  4. Click the Additional Settings tab.

  5. Click Add.

  6. On the Add Additional Settings dialog:

    • In the Name box, type sSTREAMSPOOLSIZE.

    • In the Category box, select or typeOracleAgent from the list.

    • In the Type box, select String.

    • In the Value box, type the memory for Oracle streams(for example, 96M).

  7. Click OK.

Setting the UNDO Retention Period

Whenever a transaction is committed, the old undo information is retained by default for a period of 1800 seconds. You can modify this value, using the following steps:

  1. From the CommCell Browser, navigate to Client Computers.

  2. Right-click the <Client> and then click Properties.

  3. Click Advanced.

  4. Click the Additional Settings tab.

  5. Click Add.

  6. On the Add Additional Settings dialog:

    • In the Name box, type sUNDORETENTIONSIZE.

    • In the Category box, select or typeOracleAgent from the list.

    • In the Type box, select String.

    • In the Value box, type the retention period (for example 2200 seconds).

  7. Click OK.

Viewing RMAN errors

  • If the system notifies you that there was an "RMAN error" during a backup or restore operation, we advise you to view the RMAN Output Log to identify which error was encountered. To view the RMAN Output Log, do the following:

    1. From the CommCell Browser, right-click the <Instance>, point to View and then click either Backup History or Restore History.

    2. Click OK.

    3. On the Job History window, right-click the job and then select View Log File. The RMAN Output Log will display for the job.

    Once you have identified the specific RMAN error, consult one of the following publications from Oracle Corporation for information on resolving the specific error, in conjunction with your on-site Oracle database administrator:

    • Oracle - Backup and Recovery Guide

    • Oracle - Recovery Manager User’s Guide and Reference

    • Oracle - Recovery Manager User’s Guide

CommCell Console errors

  • If only command line backups have been performed, and a Browse Restore operation is subsequently attempted from the CommCell Console without first opening and closing the subclient properties, then the system will display a message indicating that no data was backed up. If this happens, ensure that you open and then close the subclient properties of the Oracle Agent you are trying to restore, then try the Browse Restore operation again.

  • If the line "set linesize 80" is present in the file $ORACLE_HOME/sqlplus/admin/glogin.sql, it may cause the SrvOraAgent server process on the CommServe computer to fail when browsing database contents or executing a backup. To avoid such failures, comment out that line from the file and re-try the browse or backup operation.

  • For Oracle 10g and higher, if the instance is configured for autobackup with flash recovery, then restoring the SP File from the CommCell Console will not work. The following work-arounds are available in this situation:

    • Work-around 1:

      Comment out the option "db_recovery_file_dest" from the PFile, then re-create the SPFile using the new PFile. Afterwards, restoring the SP file from the CommCell Console should work.

    • Work-around 2:

      Restore the SP file manually using RMAN with the script below:

      RMAN> run {
      2> ALLOCATE CHANNEL c1 DEVICE TYPE DISK;
      3> restore spfile from autobackup;
      4> }

Point-in-time recovery

When you recover a database to a point in time, the RMAN command ALTER DATABASE OPEN RESETLOGS is executed which will reset the SCN (System Change Number) and time stamp on every object of the database (datafiles and control files). Also, only the archived redo logs that match the RESETLOGS SCN and timestamp value is applied to the database, thus recovering the database to a time that is not current. This is a very useful operation if the point-in-time to which you are trying to recover is certain and known, but can be counterproductive if you are guessing at the point-in-time.

If you are not sure about the point-in-time for the recovery, it is recommended to restore the data and the control files to a point in time without recovery. This method will allow you to restore the database to a state that you can make the determination whether or not you have achieved the correct point-in-time, without invoking the "ALTER DATABASE OPEN RESETLOGS" statement that would reset SCNs and time stamps on the database objects.

After determining the correct point-in-time through this method, you can recover the database to the point in time to reset your Oracle database to the desired incarnation.

Sample scripts are provided below for your Oracle database administrator to use as reference for developing custom scripts that you can run from the RMAN command line, to perform special operations apart from the CommCell Console.

Sample Script for Resetting a Database after RESETLOGS

The following example resets a database after performing an incomplete media recovery:

run {
allocate channel dev1 type disk;
set until logseq 1234 thread 1;
restore database skip tablespace readonly;
recover database;
sql "ALTER DATABASE OPEN RESETLOGS";
release channel dev1;
}

reset database;

Sample Script for Resetting the Database to an Old Incarnation

The following command makes an old incarnation of database PROD1 current again:

# obtain primary key of old incarnation

list incarnation of database prod1;

List of Database Incarnations

DB Key
------

Inc Key
-------

DB Name
-------

DB ID
-----

CUR
---

Reset SCN
---------

Reset Time
----------

1

2

PROD1

1224038686

NO

1

02-JUL-98

1

582

PROD1

1224038686

YES

59727

10-JUL-98

shutdown immediate;

# reset database to old incarnation

reset database to incarnation 2;

# recover it
run {
allocate channel dev1 type disk;
restore controlfile;
startup mount;
restore database;
recover database;
sql "ALTER DATABASE OPEN RESETLOGS";
release channel dev1;
}

Completed with One or More Errors

Restore jobs from Oracle Agent are displayed as "Completed w/ one or more errors" in the Job History in the following cases:

  • During a table restore, if the export or import of table fails.

  • RMAN recovery is completed, but an incorrect open mode is selected for restore.

Restore Completed with Warnings

Restore jobs from Oracle for Oracle Agent are displayed as "Completed with Warnings" in the Job History in the following case:

  • When the database restore succeeds but it fails when recovering the database or opening the database.

Oracle Errors

If you receive an Oracle error during an Oracle restore operation, we recommend that you follow procedures published by Oracle Corporation on resolving the specific error.

For example, Restoring an RMAN Backup to Another Node [ID 73974.1].

We also advise you to consult with your on-site Oracle database administrator, as needed.

ORCL0044: The Oracle Wallet May Not Open after an Offline Backup and Restore

Issue:

If the Oracle Wallet is defined in the default location (ORACLE_BASE) and there are no entries to the ORACLE_WALLET location in the sqlnet.ra file, the Oracle Wallet may not open after an offline backup and restore when either of the following is true:

  • The subclient is configured with no lights-out script (Subclient Properties dialog box, Advanced tab, Offline Arguments tab)

  • The Switch Database Mode for Restore option (Options tab) was used for the restore.

Solution

Use one of the following options to fix this issue.

  • Define the Oracle Wallet location in the sqlnet.ra file

  • Configure a post-script that opens the Oracle Wallet (Subclient Properties dialog box, Pre/Post Process tab, PostBackup Process box)

ORCL0045: RMAN Backup Jobs Run Slowly

Issue:

The RMAN jobs run slowly.

Solution

The sql statements may not be optimized. Set the Rules for the Oracle optimizer_mode Statement.

ORCL0046: A Restore from a Tape Library Hangs

Issue

A restore from a tape library hangs with the following error:

The medium is already reserved by some other job[s]

Resolution

This is due to default optimization that the software uses to enhance performance. To fix this issue, disable optimization. For more information, see Disabling Optimization for Database Restores.

Oracle 19c table-level restore fails with an OS authentication prompt for a password during export of a table

Issue

During the export of a table in Oracle 19c, a table-level restore fails because the OS prompts for a password. The following shows an example command and the resulting output:

C:\Windows\system32>e:\app\oracle\product\19.0.0\dbhome_1\\bin\expdp userid=\"/ as sysdba\"  SCHEMAS='RAJK' EXCLUDE=TABLE:\"NOT IN \(\'EMP\'\)\"  EXCLUDE=PROCEDURE  directory=tsrstdir DUMPFILE=cv_%U.dump LOGFILE=tsrstdir:cv_exp.log 
12244 2dcc  06/09 13:11:43 558103 ExecuteWinScript(5193) - Created process. Before ReadFile
12244 2dcc  06/09 13:11:43 558103 ExecuteWinScript(5204) - Received(2 bytes):
12244 2dcc  06/09 13:11:43 558103 ExecuteWinScript(5193) - Created process. Before ReadFile
12244 2dcc  06/09 13:11:43 558103 ExecuteWinScript(5204) - Received(179 bytes):Export: Release 19.0.0.0.0 - Production on Tue Jun 9 13:11:43 2020
Version 19.3.0.0.0
Copyright (c) 1982, 2019, Oracle and/or its affiliates.  All rights reserved.
Password:

Resolution

Take one of the following actions:

  • Apply the patch for Oracle bug 28707931.

  • Manually create an auxiliary instance with a connect string (user/password@service name) and provide the details in the Table Restore tab.