Restore Troubleshooting - SAP Oracle iDataAgent

Restore fails with permissions issue

Symptom

The restore fails due to issues accessing the Commvault registry, log files and base directories.

The RMAN restore fails because it cannot load the CommVault SBT Media Management library.

The SAP backint restore fails because the symbolic links are configured and correctly pointing to the Commvault backint.

Resolution

Run the Database Readiness Check.

Restore of large number of files fails for RMAN and BACKINT

Symptom

SAP Note (1647272 P.12) advises that if the SAP on Oracle database has BR*Tools version 7.20(28)-7.20(31), a RMAN interface restore of a large number of files (greater than 100) fails, and a BACKINT interface restore runs slowly . This is due to the chunking feature, where a large number of files are not processed at one time, but in groups of 100. This feature was introduced in patch 7.20(28) and existed till patch 7.20(31). It was disabled in patch 7.20(32).

Resolution

You can upgrade to BR*Tools 7.20(32) (or later) or to BR*Tools 7.40(4) to solve this issue.

Before you run the restore, disable the SAP feature that causes the failure by setting the following SAP special parameters in init<DBSID>.sap.

_rman_grp_cnt = 9999

_util_grp_cnt = 9999

After modifying the parameters, re-run the restore job.

Restore fails with character set name issue

Symptom

The restore fails due to mismatch of character set at the source location where RMAN is run and target database location.

Resolution

This is an Oracle issue. Please contact Oracle support for more information.

Restore completed with warnings

Restore jobs from SAP for Oracle iDataAgent will be displayed as "Completed with Warnings" in the Job History in the following cases:

• When the database restore succeeds but it fails when recovering the database or opening the database.

Database Copy Restores Initiated from the CommCell Console Might Go into Pending State When Using BR*Tools v7.40(43) and v7.40(44)

Issue

Database copy restore jobs that are initiated from the CommCell Console might go into pending state with BRRECOVER requesting extra logs which are not part of the backup jobs.

Example

BR0017I Offline redolog file '/oracle/SRI/oraarch/DSParch0_610_1067279100.dbf' not found
SQL>SQL> SQL> SQL> SQL> SQL> SQL> ORA-00279: change 34428818 generated at 03/15/2021 20:26:12 needed for thread 1
ORA-00289: suggestion : /oracle/SRI/oraarch/DSParch1_10_1067279100.dbf
ORA-00280: change 34428818 for thread 1 is in sequence #10
Specify log: {<RET>=suggested | filename | AUTO | CANCEL}
ORA-00308: cannot open archived log '/oracle/SRI/oraarch/DSParch0_610_1067279100.dbf'
ORA-27037: unable to obtain file status
Linux-x86_64 Error: 2: No such file or directory
set autorecovery off;
recover from '/oracle/SRI/oraarch/DSParch' database until cancel using backup controlfile;
/oracle/SRI/oraarch/DSParch0_610_1067279100.dbf
cancel
exit;
BR0338E Applying offline redolog file /oracle/SRI/oraarch/DSParch0_10_1067279100.dbf failed
BR0338E Applying offline redolog file /oracle/SRI/oraarch/DSParch0_11_1067279100.dbf failed
BR0338E Applying offline redolog file /oracle/SRI/oraarch/DSParch0_12_1067279100.dbf failed
BR0797I Number of restored/recovered database files: 18/0
BR0798I Number of restored/applied incremental files: 0/0
BR0799I Number of restored/applied archivelog files: 0/0
BR0706I End of database recovery: vfgadmrw.dpt 2021-03-16 17:16:13
BR0280I BRRECOVER time stamp: 2021-03-16 17:16:13
BR0704I BRRECOVER terminated with errors

Resolution

As a workaround, place the destination instance in shutdown mode and run the BRRECOVER command.

For more information about BR*Tools command line interface, see SAP on Oracle Restore Examples Using the BR*Tools Command Line Interface

BRRESTORE Can Cause Movement and Logging to Stop in Windows Configuration

Symptom

On Windows Configurations, when you launch BRRESTORE from the CommCell Console, or the command line interface, the following issue occurs.

There is no further movement in restore Command after lines- BR0229I Calling backup utility with function 'restore'...

There is no further logging for the Commvault logs on the client side.

Example

[brrestore -d util_file -b2 SAP_XXXX_XXXX -m G:\oracle\CER\saparch\archCER.log -c force -u / ]
 brrestore Log:[
 BR0401I BRRESTORE 7.20 (34)
 BR0405I Start of file restore: reoxjbwp.rsf 2014-10-07 10.57.43
 BR0484I BRRESTORE log file: G:\oracle\CER\sapbackup\reoxxlkm.rsf
 BR0280I BRRESTORE time stamp: 2014-10-07 10.57.44
 BR0407I Restore of database: CER
 BR0408I BRRESTORE action ID: reoxxlkm
 BR0409I BRRESTORE function ID: rsf
 BR0415I File for restore using backup utility ID SAP_XXXX_XXXX: G:\oracle\CER\saparch\archCER.log
 BR0416I 1 file found to restore, size 0.000 MB
 BR0421I Restore device type: util_file
 BR0134I Unattended mode with 'force' active - no operator confirmation allowed
 BR0100W Internal error for 'stop_file' at location BrStopCheck-1
 BR0280I BRRESTORE time stamp: 2013-09-07 10.44.44
 BR0229I Calling backup utility with function 'restore'...

Resolution

Disable the User Account Control (UAC) for the Windows user and resubmit the restore.

Restore - SAP Oracle CommCell Console Restores May Fail with the Wrong RMAN Syntax Construction in the Case of Multiple Database Resets

Symptom

The SAP Oracle CommCell Console restore may fail due to an RMAN syntax error when there are multiple database resets.

Example

run { allocate channel dsk device type disk;} 
 run { allocate channel sbt_1 device type 'SBT_TAPE' parms
 'SBT_LIBRARY=/opt/dbserve20/Commvault/Base/libobk.so,BLKSIZE=1048576,ENV=(CV_mmsApiVsn=2,ThreadCommandLine=SAPRESTORE 
 1182598 1182598 -p 2 -jt 1182598:2:1 saporavm3 -vm Instance001 -cn saporavm3 -rcp 0 -rcp 0 -rcp 0)'; 
 restore archivelog from sequence 3 until sequence 4 thread 1 from tag aeouuxih force; 
 restore archivelog from sequence 5 until sequence 5 thread 1 from tag aeouuzek force; 
 restore archivelog from sequence 10 until sequence 11 thread 1 from tag aeouvarw force; 
 release channel dsk;release channel sbt_1; } 
 exit;

Resolution

Upgrade to BRTools 7.40 patch level 12.

Restore fails when there are multiple subclients for data only and log only backups

Symptom

When there are multiple subclients for a database and one subclient is for data only backups and the other for log only backups, a data and log restore may fail if a user selects a backup job that contains only data. The restore job error is:

[Failing Restores as the log summary file information could not be found. Try restores from the command line.]::Client [sde-sap1] Application [ClSapAgent] Message Id [301989971] RCID [0] ReservationId [0]. Level [0] flags [0] id [0] overwrite [0] append [0] CustId[0]

Resolution

Perform a point-in-time restore for the backup cycle where the data and logs are available. For more information, see Restoring the Database to a Point-in-Time.

SAPOR0007: Validation Error When Running Third Party Command Line Scripts

Issue:

An error message containing "Provide Valid Token" is returned when a third party command line script runs.

Solution

A valid token file was not included in the request.

1. Run the qlogin command with the token file option (-f) to obtain a token file.

2. Use the CvQcmdTokenFile or CVOraSBTParms parameter with the token file that the qlogin command generates.

   For information on required and optional SBT parameters, see SBT Parameters.

SAPOR0008: User Error When Running Third Party Command Line Scripts

Issue:

An error message containing "Provide competent user" is returned when a third party command line script runs.

Solution

The user does not have the correct permissions in the CommCell Console to run the backup job.

SAPOR0010: Restore Disk Space Errors

You can receive one of the following errors when there is not enough disk space for a restore.

• BR0428W File will be overwritten

• BR0360E Not enough disk space

The No File Delete (NFD) option prevents BRRESTORE from deleting the files during a restore. When you use this option, BRRESTORE overwrites the files.

The No Space check (NSC) option suppresses the space error message and presents a warning message instead, allowing the restore to continue.

Warning: Only use these options when you are sure that there is enough available space for the restore.

If you performed the restore with BR*Tools, run the restore again with the NFD or NSC option.

Util_file Interface

1. On the command line, type one of the following commands.

   Restore with the NSC option.

   Brrestore –d util_file –b last –m all –NSC

   Restore with the NFD option.

   Brrestore –d util_file –b last –m all –NFD

If you performed the restore from the CommCell Console, set the Additional Setting for the option that you want to use and run the restore again. For information, see Allowing the No Space Check or No File Delete Option for BRRESTORE.

SAPORA0011: Third-Party util_file Command Line Restores Fail to Submit Restore Jobs with CommCell Console Authentication

You can receive the following error message when you run a third-party command line restore with the util_device and are using the CommCell Console third-party command line authentication. For more information, see configuring the parameters for CommCell Console third-party job authentication.

errorMsg=[Token validation failed. Please provide valid CommCell User Token file to run jobs. Third party jobs are being authenticated at CommServer]

Resolution

1. Set the file permissions on the token file so that application users can read the file.

Example

If the token file name is "tokenfile, on the command line type the following:

chmod 740 /tmp/tokenfile

SAPOR0013: BR*Tools RMAN_UTIL DATA only backup does not contain the logs for BRRECOVER

This applies to BRRECOVER command line restores only. If you have data and log backups that you performed with the RMAN_UTIL device type and if the logs that you want to recover are not in the latest cycle, then set sUSEBKPIDFORCTRLRESTORE to the complete encoded EBID of the log backup job ID that contains the control file needed for recovery.

For example, if the RMAN_UTIL backup job in the CommCell Console is 454912, then the complete encoded EBID is 36937142_454912.

Example Command

brbackup -d rman_util -t online -m full -u / -c force

Before You Begin

Set the sUSEBKPIDFORCTRLRESTORE environment variable in the database user login session before you initiate the BRRECOVER command line restore.

Procedure

Set the TRC level to 11 to verify the BRRECOVER environment.

1. On the command line, type the following command, substituting the SID parameter:

    
   brrecover -f SID -t dbpit -n last -c force -u / -TRC 11

2. Open the BRRECOVER log file and look for a line similar to the following (In this example, 36937142_454912 is the complete encoded EBID).

   'sUSEBKPIDFORCTRLRESTORE=36937142_454912'
   

You can also look in the backint_oracle.log file to see that the complete encoded EBID that you specified was used for the control file restore (the #RESTORE information).

In this example it is #RESTORED 36937142_454912.

6068 1794 11/03 17:55:16 42853 ::analyzeLine() - 2-nextFile=[G:\oracle\CER\sapreorg\CNTRLCER.DBF] destDir=[] nextLine=[36937142_454912 "G:\oracle\CER\sapreorg\CNTRLCER.DBF"] nextLine2=["G:\oracle\CER\sapreorg\CNTRLCER.DBF"] 
 6068 1794 11/03 17:55:16 42853 ::intimateSapRestore() - #RESTORED 36937142_454912 G:\oracle\CER\sapreorg\CNTRLCER.DBF

SAPOR0014: Restore of Data and Log Backups May Fail when Using a Copy Precedence

Symptom

If you browse the data and log backups by using a copy precedence, the restore fails when the data backup and log backup have different storage policies with different copy precedence values. The error is similar to the following:

Error Code: [18:72] Description: No log sequence number ranges were found. Source: dbtestlinux6, Process: ClSapAgent

Resolution

Ensure that the data and log backups are present on the same copy precedence for their respective storage policies.

Use a command line restore to perform one of the following restores:

• Select and restore the data only backup by provide the data storage policy copy precedence value

• Select and restore the log only backup by provide the log storage policy copy precedence value

SAPOR0018: BRRECOVER Fails to Apply the Log Files After a Restore

Issue

BRRECOVER fails to apply the log files after a restore. The error message looks similar to the following message:

BR0783I Archivelog files with sequence number 426-496 of database instance CES will be applied now   
 SQL>SQL> SQL> SQL> SQL> SQL> SQL> ORA-00283: recovery session canceled due to errors ORA-00264: no recovery required   
 SQL> SP2-0734: unknown command beginning "/oracle/CE..." - rest of line ignored. 
 SQL> SP2-0734: unknown command beginning "/oracle/CE..." - rest of line ignored. 
 SQL> SP2-0734: unknown command beginning "/oracle/CE..." - rest of line ignored. SQL> SP2-0734: unknown command beginning "/oracle/CE..." - rest of line ignored. SP2-0044: For a list of known commands enter HELP and to leave enter EXIT.

Resolution

Make sure that your BR*Tools have the latest path level, and then retry the restore.

SAPOR0019: SAP on Oracle Database Copy restore might fail with certain versions of BR*Tools

Issue 1

SAP Note 2250026 states that corrections in BR*Tools 7.40 Patch 18 Point 2 cause BRRECOVER to delete the database file that were just reloaded. SAP has addressed this issue in SAP BR*Tools v7.40(18) and higher.

Example

BR0790I Unneeded database file /oracle/PRD/sapdata4/sr3usr_1/sr3usr.data1 will be dropped and deleted 
 BR0675I This is a recommended action - do you want to execute it now? 
 BR0126I Unattended mode active - continuing processing with default reply 'yes' BR0792I Database file /oracle/PRD/sapdata4/sr3usr_1/sr3usr.data1 can be saved before deletion 
 BR0675I This is a optional action - do you want to execute it now? 
 BR0126I Unattended mode active - continuing processing with default reply 'no' 
 BR0793I Database file /oracle/PRD/sapdata4/sr3usr_1/sr3usr.data1 dropped successfully BR0794I Database file /oracle/PRD/sapdata4/sr3usr_1/sr3usr.data1 deleted 
 BR0480I Directory /oracle/PRD/sapdata4/sr3usr_1 deleted This error occurs only if the option "-n|-seq|-sequence" was used for the determination of the end time.

Resolution 1

SAP has addressed this issue in SAP BR*Tools v7.40(18) and higher.

Issue 2

BRRECOVER could terminate during a database copy restore with BR*Tools version v74036 or lower. The following error messages are displayed:

BR1802E Oracle spfile '$ORACLE_HOME\dbs\spfileSID.ora' not found
BR1401E Creating of control files with new database name (option '-f|-ccf|-crctlfile') is supported only if spfile is used

Resolution 2

This is an SAP BR*Tools issue and is fixed in BR*Tools version v74038 and above.

SAPOR0020: A Datafile Restore That Uses the rman_util Device Type Fails

Issue

A datafile restore that uses the rman_util device type fails with an error similar to the following error:

channel sbt_1: restoring datafile 00099 to I:\ORACLE\CER\SAPDATA3\SR3731_7\SR3731.DATA7
 channel sbt_1: reading from backup piece CER_beweiswd.865145_1 
 channel sbt_3: ORA-19870: error while restoring backup piece CER_beweiswd.865143_1 
 ORA-19501: read error on file "CER_beweiswd.865143_1", block number 1 (block size=512) 
 ORA-27190: skgfrd: sbtread2 returned error 
 OSD-04006: ReadFile() failure, unable to read from file 
 O/S-Error: (OS 1191) The system shutdown cannot be initiated because there are other users logged on to the computer. 
 ORA-19511: Error received from media manager layer, error text channel sbt_3: starting datafile backup set restore 
 channel sbt_3: specifying datafile(s) to restore from backup set 
 BR0280I BRRESTORE time stamp: 2015-07-05 11.49.11

Resolution

Completely disable the Windows User Acess Control (UAC), and then resubmit the restore.

SAPOR0021: A Database Copy Initiated from the CommCell Console Might Go to the Pending State

Database Copy Uses the RMAN_UTIL Device and There Are More than 100 Log Files to Restore

Issue

A database copy that meets the following conditions goes into a pending state:

• The database copy was initiated from the CommCell Console.

• There are more than 100 log files that need to be restored.

• The device type is RMAN_UTIL

Resolution

Use one of the following workarounds:

• Use the BRRECOVER command to perform the restore. For more information, see BR*Tools Example: Performing a Cross-Machine Restore to the Same SAP on Oracle SID or BR*Tools Example: Performing a Cross-Machine Restore to a Different SAP on Oracle SID Using the BRRECOVER Command.

• Use the BRRESTORE command to perform the restore. For more information, see BR*Tools Example: Performing a Cross-Machine Restore to a Different SAP on Oracle SID Using the BRRESTORE Command.

Database Copy Uses the RMAN_UTIL or UTIL_FILE Device and There Are More than 1000 Log Files to Restore

Issue

A database copy that meets the following conditions goes into a pending state:

• The database copy was initiated from the CommCell Console.

• There are more than 1000 log files that need to be restored.

• The device type is RMAN_UTIL or UTIL_FILE

Resolution

Use one of the following workarounds:

• If you are performing a database copy from the CommCell Console, then on the SAP Oracle client set the nBRRECOVER_NUMOFLOGSTORESTORE additional setting to the number of logs, and then retry the operation.

• Use the BRRECOVER command with the -AHC option. On the command line, type the following command:

   
  brrecover -d device_type -c force -t dbpit -b BRBACKUP_detail_file -f SID -seq last -AHC 10000

59441: Database copy restore fails during recover phase

For more information, see 59441: SAP for oracle database copy restores fail during the recover phase.

SAPORA0022: Database Copy Might Fail when The Source Client is Version 9 or Version 10

Issue

A database copy copy might fail under the following conditions:

• The database copy source client runs Commvault Version 9 or Version 10 software

• The destination client runs Commvault Version 11

• The database copy wa performed from the CommCell Console

• The device type is rman_util

  The error looks similar to the following error:
  Error message Use backupStamp=[\#NULL] for control fle restore
  BackupId not specified - Illegal latest BackupId obtained=<\#NULL>
  

Resolution

When you perform the restore, make sure that you select the following items to include in the restore:

• The data

• The log files

• The control file

SAPORA0023: Database Copy Fails with BR0156E Error

Issue

If you have BR*Tools v7.40 patch level 1 or earlier, a Database Copy that you perform from the CommCell Console fails with an error similar to the following error:

BR0156E Illegal value '2018-05-04 05:37:03' found for option '2018-05-04'

Resolution

The time format changed in the BR*Tools versions. For more information, see Remark 2 in SAP Note 1838850.

To resolve this issue, upgrade to BR*Tools v7.40 path level 2 or later, and then retry the operation.

You can also perform a SAP Oracle BR*Tools cross-machine restore from the command line. For more information, see BR*Tools Example: Performing a Cross-Machine Restore to a Different SAP on Oracle SID Using the BRRECOVER Command.

SAPORA0024: Database Copy Fails with a Wrong Control File Path

Issue

With BR*Tools version 7.20 (42), the BRRECOVER log adds an extra folder name in the control file path with database copy.

So, database copy operations that include a control file restore, fail with a wrong control file path.

BRRECOVER summary file log

BR0280I BRRECOVER time stamp: 2018-11-23 11.38.34
BR0134I Unattended mode with 'force' active - continuing processing with defaultcreply 'cont'
BR0200I BR_TRACE: location BrUtilCall-5, backup utility input file: /oracle/TST/sapbackup/.vezwmldp.lst
'#NULL /oracle/PRD/121/sapreorg/cntrlPRD.dbf /oracle/TST/sapreorg

backint_oracle.log

'Success in getting Job id=937266 Token=<>^'
'AppId=247 JobId=784728 App=<2:247> subClientName=<default>^'
'^'
'#NOTFOUND /oracle/PRD/121/sapreorg/cntrlPRD.dbf^'
BR0386E No backup of file '/oracle/PRD/121/sapreorg/cntrlPRD.dbf' found by backup utility
'Could not find the file... </oracle/PRD/121/sapreorg/cntrlPRD.dbf>^

Resolution

Update the BR Tools version to 7.40 (16) or higher.

SAPORA0024: BRRECOVER Process Fails

Issue

When the online redo logs are on different path on the source and destination clients, during a restore, the Control File creation process looks for the logs on the same path as the source, and then fails the restore operation with the following error message:

BR1406E Creation of new control files failed

Resolution

The Online Redo log path should be directly under SAPDATA_HOME directory on the source and destination clients, or make sure that the same path on the source client also exist on the destination client prior to running the database copy restores. This is a SAP requirement.

SAP Oracle restores fail with no backup error

For more information, see KB article SAP Oracle restores fail with no backup error.

Oracle Errors

If you receive an Oracle error during a SAP Oracle restore operation, we recommend that you follow procedures published by Oracle Corporation on resolving the specific error. We also advise you to consult with your on-site Oracle database administrator, as needed.

The SAP for Oracle RMAN restore job fails during the log restore phase

For more information, see KB article SAP for Oracle RMAN restore job fails during the log restore phase.