SAPOR0003: UTIL_FILE_ONLINE backup operation fails on Linux
The UTIL_FILE_ONLINE backup operation hangs on Linux and fails after 30 minutes of wait time.
Symptom
Switch files are created at the default location $ORACLE_HOME/sapbackup and are used for communicating between BRTOOLS and BACKINT.
After the backup is initiated, BRBACKUP could not find the switch files at the default location as BACKINT creates them at a new location $SAPDATA_HOME/sapbackup. This causes a lack of communication between BRTOOLS and BACKINT, hence delaying the backup operation which eventually fails after 30 minutes of waiting.
Resolution
Create a symbolic link from $ORACLE_HOME/sapbackup to $SAPDATA_HOME/sapbackup. This will ensure the expected communication between BRTOOLS and BACKINT to complete the backup operation successfully.
SAPOR0005: BR*Tools do not have the privilege to bring database down for a backup
Symptom 1
The following error message is displayed when you run an offline backup.
ERROR CODE [18:95]: SAP Oracle Error [SQL error -1033 at location BrDbConnect-2, SQL statement:'CONNECT /' ORA-01033: ORACLE initialization or shutdown in progress Detail File=[/oracle/CER/sapbackup/beqiunem.anf] ].Resolution
Resolution
Verify that the instance exists in the CommCell Console, with the following parameters configured.
-
On the General tab of the Instance Properties dialog box:
-
In the User Account box, the login credentials to access the SAP for Oracle client.
For UNIX configurations, use ora(dbsid) for this value.
For Windows configurations, use <domain_name>\ora(dbsid) for this value.
-
In the ORACLE HOME box, the SAP on Oracle application install path.
Use the following procedures to obtain the Oracle home path.
UNIX
Log on to the SAP on Oracle server with the ora(dbsid) user.
On the command line, type the following command:
su - ora(dbsid) echo $ORACLE_HOME
Windows
Log on to the SAP on Oracle server with the <domain_name>\ora(dbsid) user.
On the command line, type the following command:
echo %ORACLE_HOME%
-
In the SAP DATA PATH box, the SAP on Oracle data and log file path.
The SAP server environment variables contain this information.
Use the following procedures to obtain the SAP data path.
UNIX
Log on to the SAP on Oracle server with the ora(dbsid) user.
On the command line, type the following command:
su - ora(dbsid) echo $SAPDATA_HOME
Windows
Log on to the SAP on Oracle server with the <domain_name>\ora(dbsid) user.
On the command line, type the following command:
echo %SAPDATA_HOME%
-
In the SAP EXE PATH box, the SAP on Oracle executable files.
Use the following procedures to obtain the SAP data path.
UNIX
Log on to the SAP on Oracle server with the ora(dbsid) user.
On the command line, type the following command:
su - ora(dbsid) echo $SAPEXE
Windows
Log on to the SAP on Oracle server with the <domain_name>\ora(dbsid) user.
On the command line, type the following command:
echo %SAPEXE%
Symptom 2
The BR*Tools must be able to bring down the SAP on Oracle database for a backup.
For BRBACKUP to run offline backup, OPS$external user should have the SYSOPER & SYSDBA privilege.
Resolution
On Windows:
Add the OS Database Administrators with the Local and Domain Administrator user on the "Oracle Administration Assistant for Windows" wizard.
On Unix:
When a DBA allows external authentication for an Oracle database, they must create the corresponding OPS$<username > in the database and then grant dba level privileges.
Run the following query.
Sql> grant dba to opsSIDadm;
SAPOR0006: Backups Fail in a Windows Environment with Multiple Oracle Installs or Oracle Instances
Symptom
The backup fails in Windows environments that have multiple Oracle instances or multiple Oracle installs. For additional information, go to the SAP on Oracle Community Network site (scn.sap.com/welcome) and search for "SAP Note 556232 - Environment settings for R/3/Oracle on Windows".
Resolution
When you use several databases on the same server, you cannot set the following Oracle variables on the SYSTEM environment. You must set them in the database user environment.
-
LOCAL
-
ORA_NLS33
-
DIR_LIBRARY
-
auth_shadow_upgrade TNS_ADMIN (except with R/3 3.1I or lower or Oracle 10g)
-
TWO_TASK
SAPOR0007: Validation Error When Running Third Party Command Line Scripts
Issue:
An error message containing "Provide Valid Token" is returned when a third party command line script runs.
Solution
A valid token file was not included in the request.
-
Run the qlogin command with the token file option (-f) to obtain a token file.
-
Use the CvQcmdTokenFile or CVOraSBTParms parameter with the token file that the qlogin command generates.
For information on required and optional SBT parameters, see SBT Parameters.
SAPOR0008: User Error When Running Third Party Command Line Scripts
Issue:
An error message containing "Provide competent user" is returned when a third party command line script runs.
Solution
The user does not have the correct permissions in the CommCell Console to run the backup job.
SAPOR0009: The BRBACKUP command with the -u / Option or SQLPLUS / fails
Issue
The BRBACKUP command with the -u / option or SQLPLUS / fails.
Connect String Considerations
The software uses the Connect String that you specify for the instance for browse operations and to view the instance status. Depending on your configuration, the SAP for Oracle agent uses “/” or “//” for data protection and recovery operations. If you want to use “//”, then you must configure the SAP secure store on Commvault and on SAP. For more information on SSFS, go to the SAP support site and see SAP Note 1611877 Support for ABAP SSFS during database connect.
To use “system” as the user, the password must be "manager" and you need to set the following additional setting. To the SAP Oracle client, add the additional setting as shown in the following table.
For instructions on how to add additional settings from the CommCell Console, see Adding or Modifying Additional Settings from the CommCell Console.
Property |
Value |
---|---|
Name |
|
Category |
OracleSapAgent |
Type |
String |
Value |
Y |
Note:
If you disabled OS authentication, and the system user password is not "manager" and you do not have the SSFS feature enabled, you can use the Oracle connect string specified in the SAP Oracle instance properties to run the brbackup, brarchive, brrestore, and rerecover commands.
To use the connect string, set the bUseConnectStringForBRTools additional setting. To the SAP Oracle client, add the additional setting as shown in the following table.
For instructions on how to add additional settings from the CommCell Console, see Adding or Modifying Additional Settings from the CommCell Console.
Property |
Value |
---|---|
Name |
|
Category |
OracleSapAgent |
Type |
Boolean |
Value |
True |
Resolution
Windows Configuration
-
Create a user called OPS$<domain account> if you use a domain account, or create a user called OPS$<local admin> if you use a local account.
-
Grant the dba privilege to the OPS$<domain account> user if you use a domain account, or the OPS$<local admin> user if you use a local account.
-
Shutdown the Oracle database.
-
If the "os_authent_prefix=OPS$" entry does not exist in the init<SID>.ora file, add the entry and save the file.
-
Start the Oracle database.
-
Connect to the Oracle database by using the run sqlplus / command.
UNIX Configuration
-
Create a user "OPS$<application_user>" that is externally identified.
-
Grant the dba privilege to the "OPS$ORACLE" user.
-
Commit the changes.
-
Connect to the Oracle database by using sqlplus /.
When SQLPLUS / works, the BRBBACKUP command with the -u / option will work.
SAPOR0012: Configuring an Instance or a Backup fails on Windows clients
Issue
Configuring an instance or a backup fails on Windows Clients.
Note: when using Oracle 12c, grant full control permission for the Oracle home user for the Commvault folder.
Resolution
Make sure that the Oracle user is part of administrator group and ora_dba group. If the user is not part of administrator group, assign permissions for the user as follows:
-
Grant full control permission for the ContentStore folder.
-
From Windows Explorer, right-click the ContentStore folder and select Properties.
-
Click the Security tab.
-
Select the user and click Edit or click Add if the user does not exist.
-
Select the Allow check box for Full Control permission for the user and click OK.
-
-
Grant Full control permission for the CommVault registry:
-
From the Registry Editor, go to HKEY_LOCAL_MACHINE | SOFTWARE.
-
Right click CommVault Systems and select Permissions....
-
Select the user and click Edit or click Add if the user does not exist.
-
-
When UAC is enabled, add the user to the Local Security Policy:
-
From Local Security Policy, navigate to User Right Assignment.
-
Right-click Impersonate a client after authentication and then select Properties.
-
Click on Add User or Group and then click OK.
-
SAPOR0015: Backups May Not Run if the SAPSID ADM User Runs a Backup Instead of the ORASID User
Issue
When you run a command or a script that has the SUID set, the effective UID is the owner of the file, and not the user that runs the script. The owner’s execute field displays an “s” as the value. The SGID permission is similar because it runs as if it were a member of the group and displays an “s” in the group execute field.
In Solaris 10 environment, the following behavior occurs when you run a SUID command or script. The SOLARIS linker tries to stop malicious dependency substitution by refusing to search what it considers non-secure locations. If you set the SUID and the SGID for the ORASID user for the SAPEXE folder, then the Oracle installation user can run BRBACKUP and BRBARCHIVE because the real and effective users match.
Example
You can use the UNIX list dynamic dependencies (ldd) command to determine the shared libraries for brbbackup for the ORASID user. In this example, the SID is "TRD". In this case, look for "libnnz10.so".
ldd -s /usr/sap/TRD/SYS/exe/run/brbackup
/usr/sap/TRD/SYS/exe/run/brbackup search path=/orahome/app/oracle11g/product/dbhome_1/lib:/orahome/app/oracle11g/product/dbhome_1/lib32:/orahome/app/oracle11g/product/dbhome_1/bin:
/orahome/app/oracle11g/product/dbhome_1/lib:/usr/bin:usr/sbin:/lib:/usr/sap/TRD/SYS/exe/run:/usr/bin
(LD_LIBRARY_PATH)
trying path=/orahome/app/oracle11g/product/dbhome_1/lib/libnnz10.so
libnnz10.so => /orahome/app/oracle11g/product/dbhome_1/lib/libnnz10.so
You can use the UNIX list dynamic dependencies (ldd) command to determine the shared libraries for brbbackup for the SAPSID ADM user. In this example, the SID is "TRD". In this case, look for "libnnz10.so".
/usr/sap/TRD/SYS/exe/run/brbackup
search path=/orahome/app/oracle11g/product/dbhome_1/lib
(LD_LIBRARY_PATH)
ignore path=/orahome/app/oracle11g/product/dbhome_1/lib (insecure directory name)
search path=/oracle/client/10x_64/instantclient:/oracle/client/10x_64/lib:/oracle/db_sw/102_64/instantclient:/oracle/client/instantclient:/oracle/db_sw/102_64/software/lib (RPATH from file /usr/sap/TRD/SYS/exe/run/brbackup)
trying path=/oracle/client/10x_64/instantclient/libnnz10.so
trying path=/oracle/client/10x_64/lib/libnnz10.so
trying path=/oracle/db_sw/102_64/instantclient/libnnz10.so
trying path=/oracle/client/instantclient/libnnz10.so
trying path=/oracle/db_sw/102_64/software/lib/libnnz10.so
search path=/opt/simpana/Base64:/lib/64:/usr/lib/64:/opt/snapprotect/Base:/export/home/simpana2/simpana2/Base64:/export/home/sc8/commvault2/Base64:/export/home/upg/simpana3/Base64 (configuration default - /var/ld/64/ld.config)
trying path=/opt/simpana/Base64/libnnz10.so
trying path=/lib/64/libnnz10.so trying path=/usr/lib/64/libnnz10.so
trying path=/opt/snapprotect/Base/libnnz10.so
trying path=/export/home/simpana2/simpana2/Base64/libnnz10.so
trying path=/export/home/sc8/commvault2/Base64/libnnz10.so
trying path=/export/home/upg/simpana3/Base64/libnnz10.so
libnnz10.so => (file not found)
Resolution
If you want to use the SAPSID ADM user, use the Oracle crle command to add all required search paths for the default search and trusted paths.
For information on the crle command, go to Oracle Solaris man page section, crle.
SAPOR0016: SAP on Oracle command line operations do not invoke the backint program in a Windows environment
Issue
The SAP MaxDB and SAP on Oracle command line backup and restore operations do not invoke the backint program for the Commvault software in a Windows environment.
Resolution
-
Verify that the Commvault software installation path includes "Base" in the path. On the command line, type the following command:
C:\Users\admin>echo %PATH% %JRE_HOME%\bin;C:\Program Files\Java\jreXX.xx\bin;C:\Program Files\CommVault\ContentStore\Base
where XX.xx is the Java version.
-
Verify that the backint program is searchable from ENV. On the command line, type the following command:
C:\Users\admin>where backint C:\Program Files\CommVault\ContentStore\Base\backint.exe
-
Verify that you can invoke the backint program and the output is similar to the following.
C:\Users\admin>backint backint -u SID [-f backup | restore | inquire | delete] [-t file | file_online] [-p par_file] [-i in_file] [-o out_file] [-c] [-v] Default: -f backup, -t file backint -u SID <-f summary> -p par_file [-i in_file] [-o out_file] [-v] backint -f createsldxml short_client_name oracle|maxdb
-
We recommend that you do not install multiple vendor backup software on the same machine that has the backint program.
SAPOR0017: An online_cons Command Line Backup Job Is Split into a Data Command Line and a Log Command Line Job
Issue
An online_cons command line backup job that uses a rman or util backup device is split into a data command line job and a log command line job.
Resolution
The instance that is associated with the subclient does not have a value for the SAP on Oracle $SAPDATA_HOME environment path. Make sure that you have the path specified for the instance. For more information, see Adding a SAP for Oracle Instance.
The SAP server environment variables contain this information.
You can use the following commands to obtain the SAP data path.
UNIX
Log on to the SAP on Oracle server with the ora<SID> user, substituting the SID parameter with the Oracle SID of your database.
On the command line, type the following command:
echo $SAPDATA_HOME
Example
su - oraTRD
echo $SAPDATA_HOME
/oracle/TRD
Windows
Log on to the SAP on Oracle server with the <domain_name>\ora<SID> user, substituting the domain_name parameter with your database domain name and the SID parameter with the Oracle SID of your database.
On the command line, type the following command:
echo %SAPDATA_HOME%
Example
echo %SAPDATA_HOME%
I:\oracle\TRD
SAPOR0018: A SAP Oracle Log Backup Fails or Goes to a Pending State
Issue
A SAP Oracle log backup might fail or go to a pending state because BRARCHIVE continuously runs. The error is similar to the following error:
sbtbackup: Job[xxxxx] thread[2904]: Connection with ClOraAgent failed(180 secs) for channel=3.
Resolution
When BRARCHIVE runs, the program checks if SAP Oracle created new archive logs so that the archive logs are saved in the next cycle. This operation can cause BRARCHIVE to continuously run.
To prevent this, set the _one_save_cycl parameter in the init<SID>.sap file to yes so that BRARCHIVE is permitted one backup cycle.
Example
_one_save_cycle = yes
For more information, go to the SAP support web site, Note 214370.
SAPORA0019: A Commvault Version 10 or Version 9 Client Third-Party Command Line Backup Fails
Issue
A third-party command line backup (BRBACKUP/BRARCHIVE) that is performed on a Commvault Version 9 or version 10 client fails during the config phase with index errors when the following conditions are met:
-
There are multiple MediaAgents involved
-
The MediaAgents run Commvault Version 11 Service Pack 11 software
Resolution
Set the Allow load-balance on multiple data paths within one backup job parameter to 0 (Media Management Configuration dialog box, Resource Manager tab). For more information, see Media Management Configuration Parameters.
SAPORA0020: CommCell Console Backups Might Fail after an Upgrade to Oracle 12.2
Issue
After you update the SAP Oracle database to Oracle 12.2, backups that you perform from the CommCell Console might fail.
Note: Third-party command line backups are not affected.
Resolution
Recreate the symbolic link for the ORACLE_HOME directory on the SAP Oracle client, and then retry the operation.
If this does not solve the problem, then you need to investigate the messages in the log files.
SAPORA0021: Automatic Log Backup (OSC) Schedules Start a Backup when Conditions Are Not Met
Issue
Automatic log backup schedules (OSC) start a backup even when the required conditions are not met.
Resolution
If you created clones from the primary client, then you need to check if Commvault services are running on the cloned VM.
If the services are running, then you need to disable the services even when the VM is not configured in the CommCell Console.
Incorrect Number of Bytes Error for Third-Party Command Line Backup Operations
Issue
For the SAP Oracle Agent third-party command-line backups,the default value for the blocksize in RMAN_PARMS is set as 256KB when there is no value for the parameter in the initSID.sap file.
If you run a third-party command line backup operation using the RMAN_UTIL device without setting a value for the blocksize in RMAN_PARMS then the default of 256KB is used. When you restore the job from the CommCell Console, the job fails (Note, if you perform a third-party command line restore operation, the job is successful) with the following error.
Err:Did not read correct number of bytes, expected: 1048576, read: 262144.
Cause
When you restore the job from the CommCell Console, the software sets the blocksize parameter value to 1MB when there is no value set in the initSID.sap file. The job fails because the blocksize in the restore operation (1MB) has a different value than the blocksize that was used for the backup operation (256KB).
Resolution
You can run third-party command line restore operations. Verify that the blocksize parameter is not set in the RMAN_PARMS in init<SID>.sap file, and then perform the restore operation. When you do not have a value set in the RMAN_PARMS in init<SID>.sap file, the third-party command uses the default (256KB). This value matches the value that was used for the backup operation.
Backup Failure
From CommCell Console due to Invalid User Permissions
Make sure to use the following user accounts when creating SAP for Oracle instances:
For Unix clients:
SID_nameadm
For Windows clients:
client_name/SID_nameadm
On AIX when the CIO option is turned on
Issue:
The online backup of an Oracle database does not work with JFS2 and Oracle 10g .
You receive various errors when using the external tools to access a file.
cp: A system call received a parameter that is not valid.
0653-902 Cannot open the specified file for reading.
DBV-00100: Specified FILE not accessible errno(22) A system call received a parameter that is not valid.
Additional Terminology
BR*Tools, brbackup FILESYSTEMIO_OPTIONS cio BKI4008E dbv cp dd jfs2 AIX 5.3 6.1 O_CIO O_CIOR
Reason and Prerequisites
Beginning with Oracle 10g, the Oracle database server uses the advanced I/O capabilities of JFS2. If the file system type detected by the Oracle database is JFS2, the Concurrent IO (CIO) option is used to access all Oracle datafiles. The CIO option results in enhanced Oracle IO performance because it bypasses the AIX system cache.
Set the FILESYSTEMIO_OPTIONS=SETALL initialization parameter in the spfile to use CIO with JFS2 for Oracle 10g / 11g together with ASYNC IO operation. This is the default mode for JFS2 file systems with Oracle 10g.
Datafiles opened in CIO mode by Oracle using O_CIO, are prohibited by the AIX OS from being accessed in the regular file system cache by tools (for example cp). Do not open Oracle datafiles using the CIO option, or the tools receive an error from the AIX operating system when trying to access the datafiles on an open Oracle database. Use the following solutions to prevent backups from failing.
Solution
AIX 5.3 or ( AIX 6.1 with Oracle < 11.2.0.2):
Mount all JFS2 file systems containing Oracle datafiles (sapdata directories) with the mount -o cio option, which forces a bypass of the file system cache. Use the mount -o cio option for Oracle datafiles, including the Oracle redo log that are on JFS2. Access to the Oracle datafiles by all other programs which do not support CIO may be impacted, because caching and read ahead are no longer available.
Reduce the performance impact during the database backup by using the dd command with a large BLOCKSIZE parameter as shown in the initSID sap file example.
tape_copy_cmd = dd
disk_copy_cmd = dd
dd_flags = "bs=1024k"
dd_in_flags = "bs=1024k$"
compress_cmd = "dd bs=$[1024k if=$ | ( compress -c > $ ) 2>&1"
Example restore:
uncompress_cmd = "uncompress -c $ | dd bs=1024k of=$ 2>&1"
Example verify:
uncompress_cmd = "uncompress -c $ > $"
Example "compress-only" (option "-k only"):
compress_cmd = "dd bs=1024k if=$ | compress -c > $
Alternatively, use
-
the BACKINT backup utility which supports CIO.,
-
Oracle RMAN, controlled by BR*Tools.
With Oracle RMAN as backup utility, the mount-option -cio for JFS2 is not required when mounting file systems holding Oracle datafiles.
Do not mount JFS2 file systems holding $ORACLE_HOME and log files with the -o cio option.
>=AIX 6.1 with Oracle >= 11.2.0.2:
The open flag O_CIOR option introduced in AIX 6.1 allows open calls without CIO. Tools (for example cp) can access the database files in read only mode Oracle 11.2.0.2 opens the JFS2 files using the O_CIOR option when it detects AIX 6.1. Do not use the –o cio option to mount the file system. Remove the mount option "-o cio" option on Oracle 11.2.0.2/AIX6.1 to avoid the error below.
cp: A system call received a parameter that is not valid.
0653-902 Cannot open the specified file for reading.
">DBV-00100: Specified FILE not accessible
errno(22) A system call received a parameter that is not valid.
Shared Memory Error
Issue:
The backup failed because the shared memory on the HP-UX PA-RISC client has not been configured per operational guidelines.
Resolution:
Add the DisableIPC_GLOBAL file in the /apps/commvault/Base directory on the client where the backup failed.
-
Stop the Commvault software.
-
Create an empty file called DisableIPC_GLOBAL in the /apps/commvault/Base directory. From the command line, enter the following:
touch /apps/commvault/Base/DisableIPC_Global
-
Restart the Commvault software.
Backup Fails with Permissions Issue
Issue:
The backup fails due to issues accessing the Commvault registry, log files and base directories.
The RMAN backup fails because it cannot load the CommVault SBT Media Management library.
The SAP backint backup fails because the symbolic links are configured and correctly pointing to the Commvault backint.
Solution
Run the Database Readiness Check.
Backups from the CommCell Console are successful but third party Command Line backups fail
Verify if the information in the parameter files is correctly provided. On a Windows platform, the Oracle database init<SID> files may be located in $ORACLE_HOME\database folder.
In $ORACLE_HOME/dbs/init<SID>.utl, verify the following and correct the information, if required:
CvInstanceName <Name of the instance>
CvClientName <name of client>
numStreams <number of streams>
To learn more about how to run third party command line backups for the SAP for OracleAgent, see BR*Tools Backups.
RMAN third party Command Line backups are not running
Before you run backups from the RMAN command line for the SAP for Oracle Agent, set the SBT_LIBRARY path and environment variables for CvClientName and CvInstanceName in the RMAN script. For example, on a Solaris client, provide the following path:
util_par_file = ORACLE_HOME/dbs/init@.utl
rman_parms="BLKSIZE=1048576,SBT_LIBRARY=/opt/commvault/Base64/libobk.so,ENV=(CvClientName=sunsign,CvInstanceName=Instance001)" rman_channels=1
where Cvclientname and CvInstancename are the names of the client and instance (for example, Instance001) where the SAP for Oracle Agent is installed.
On a Windows client, edit the $ORACLE_HOME\database\init<SID>.sap file and provide the parameter as given below
util_par_file = ORACLE_HOME\database\init@.utl
RMAN_PARMS="SBT_LIBRARY=,BLKSIZE=1048576,ENV=(CvClientName=client,CvInstanceName=client_name)"
where Cvclientname and CvInstancename are the names of the client and instance (for example, Instance001) where the SAP for Oracle Agent is installed.
The SBT_LIBRARYfor the various platforms are listed below:
Platform |
SBT_LIBRARY |
---|---|
AIX with 64 bit Oracle |
<Client Agent Install Path>/Base/libobk.a(shr.o) |
HP UX PA RISC 64 bit Oracle |
<Client Agent Install Path>/Base64/libobk.sl |
Solaris with 64 bit Oracle |
<Client Agent Install Path>/Base64/libobk.so |
All Other Unix platforms |
<Client Agent Install Path>/Base/libobk.so |
NOTE: The SBT_LIBRARY parameter is not applicable on Windows platforms.
When you use the RMAN utility on Solaris client, set the following parameter on the client computer:
crle -64 -c /var/ld/64/ld.config -l/opt/commvault/Base64:/lib/64:/usr/lib/64
When you use RMAN_UTIL after the logs have been reset, use the util_file interface to backup all the archive logs. Subsequent log backups can be performed using the RMAN_util interface.
To learn more about how to run third party command line backups for the SAP for OracleAgent, see BR*Tools Backups.
Jobs Completed with Warnings
A job is marked as Completed with Warnings [CWW], if Brbackup or Brarchive returns warnings during backups or restores or the database restore succeeds, however the database could not be opened or recovered successfully.
The SAP for Oracle backup may complete with a warning. This error may occurred if the SAPSECULIB (libsapsecu.so) library cannot be found or the permissions are not correct. To correct the problem, complete the following steps:
-
Ensure that the permissions for the br* executables are correct.
-
Ensure that the SHLIB_PATH and DIR_LIBRARY environment variables are set correctly.
-
On Unix systems, ensure that the DIR_LIBRARY environment variable is set on the directory that contains the libsapsecu library; for example:
DIR_LIBRARY = /usr/sap/<SID>/SYS/exe/run
This is especially important if you use an external scheduler (for example, cron, at, or BACKINT scheduler) to start the BR*Tools. Also, ensure that DIR_LIBRARY is set for both user ora<sid> and <sid>adm.
-
Ensure that the libsapsecu.so library is accessible.
-
Ensure that the SAPSYSTEMNAME environment variable is set correctly.
-
Ensure that the contents of the database (i.e., SAP tables and SAP owner) are accessible. This is especially important if you use an external scheduler (for example, cron, at, or BACKINT scheduler) to start BR*Tools. In such a case, the SAPSYSTEMNAME variable must be set in the environment of this scheduler (for example, in an environment file); in this case, it must be set within the backup profile of Commvault (consult your backup vendor).
-
Verify this setup by appending "-TRC11" to the brbackup command line that is called by Commvault.The created trace file will contain all the active environment variables at runtime of the brbackup command.
There could be other scenarios where a job could complete with a warning. For example, if logs were prematurely deleted without backing up, etc. SAP detail file located in the SAPBACKUP/SAPARCH directory should be reviewed for more details.
Oracle Errors
If you receive an Oracle error during a SAP for Oracle backup operation, follow the procedures published by Oracle Corporation on resolving the specific error. We also advise you to consult with your on-site Oracle database administrator, as needed.
General Performance Tuning and Troubleshooting
If you are experiencing performance issues during a backup, you can troubleshoot them by enabling logging of performance details in the log files. These performance counters contain information that help in resolving the performance related issues during backups.
-
Perform a client backup to determine the performance statistics. To perform a backup, see Performing a Full Backup.
You can track the progress of the job from the Job Controller window of the CommCell Console.
-
Right-click the backup job and click Details and verify the Data Transferred on Network.
-
For example, if backup job is using 10 streams, make sure to backup at least 200 GB of data. If you are performing backups using 5 streams, make sure to backup at least 100 GB of data.
-
If the backup transfer rate is very slow, then kill the job by right-clicking the backup job and then click Kill.
-
-
View log files of backup job to verify performance counters. See Viewing Log Files for Completed or Failed Jobs for more information.
Verify the following the performance counters tin the log files:
Total Oracle I/O Time
Time spent per SBT thread for reading the data from disk.
Total MA I/O Time
Time spent during data transfer to MediaAgent i.e., data read from the network buffer and written to the disk.
-
In the log file verify the above performance counters.
If the Total Oracle I/O Time value is more than the Total MA I/O Time value then perform the following to improve performance:
-
Verify Oracle application compression. If it is ON turn OFF the compression.
-
Verify Commvault compression. If it is ON turn OFF the compression from instance and storage policy copy level. See Setting Up Data Compression for step-by-step instructions.
-
Depending upon your environment, modify Data Files per BFS (value to 4 or 8) and Max Open Files. See Performance Tuning.
If the Total Oracle I/O Time value is lesser than the Total MA I/O Time value then perform the following to improve performance:
-
If the write throughput of the disk is slow, run CvDiskPerf tool to measure the throughput for the disk. See Disk Performance Tool for more information.
-
If the data transfer on the network is slow or you have a low bandwidth network environment, then verify Network Throughput by running CvNetworkTestTool tool. If network throughput is low then enable nNumPipelineBuffers additional setting to increase the data transfer throughput from the client. See Increasing Data Transfer Throughput From Client for more information.
-
Performance Tuning for RMAN_UTIL Interface
To enhance the performance for backups using RMAN_UTIL, edit the
$ORACLE_HOME/dbs/init<SID>.sap or on a Windows platform,
$ORACLE_HOME\database\init<SID>.sap file
Modify the values for the following parameters:
-
BLKSIZE
-
Channels
-
filesperset
-
maxopenfiles
For example:
rman_parms="BLKSIZE=1048576 SBT_LIBRARY=software_installation_directory/Base/libobk.so,ENV=(CvClientName=client1,CvInstanceName=Instance00x)" rman_channels=3 rman_filesperset=8 rman_maxopenfiles=2
*You do not need to provide the SBT_LIBRARY PATH parameters on Windows platforms. Please refer to SBT_LIBRARY PATHS for various platforms.
Checklist for General Troubleshooting
In case you encounter errors while performing backups, you can perform the following mentioned general troubleshooting steps to resolve the issue:
-
Verify that the SAP for Oracle Agent instance properties are correct.
-
Verify if the path of SAPEXE file is correct on the client computer.
-
Verify if the parameter files (init<SID>.ora, init<SID>.sap, init<SID>.utl) have correct information.
-
Verify the SAP user profile for correctness.
-
Increase the debug level for the below log files present on the client computer. For steps to set debug level for a log file, see Configuring Log File Settings in the CommCell Console.
-
ClSapAgent.log
-
backint_oracle.log
-
ORASBT.log
-
-
User should have the permission to SAPEXE and SAPDATA_HOME.
-
Commvault group should be associated during the SAP for Oracle Agent installation.
-
SAP BR detail file and Summary file should be present an the following paths:
-
$SAPDATA_HOME/sapbackup
-
$SAPDATA_HOME/saparch
-
-
On the Windows platform, verify if the SAP service user has been explicitly assigned full control permissions to the Commvault directory and the client registry.
Job completed with Errors
A job is marked as Completed with Errors [CWE] if some data is backed up or restored and then the job is killed or fails.
SAP for Oracle backups fail on the Windows computer when the environment variable path exceeds the maximum limit
For more information, see KB article 77159.
SAP for Oracle backups using util_file_online device fails if the SAP backup directory does not exist under SAP data home path or if the write permissions are missing on SAP backup directory
Set the SAPSWITCH environment variable for any existing directory where SAP can create the required .switch files in the system environment of the SAP Oracle application user, and restart the Commvault services. This is helpful for util_file_online device jobs.
Sample example
On Windows environment
Under system variables, add new variable follwing the example mentioned below. Modify the paths according to user environment.
SAPSWITCH=I:\Sapswitch
When util_file_online device job runs in the SAP backup detail file, the environmental variable appears as shown below:
br_env SAPSWITCH=I:\Sapswitch
On Unix environment
Set SAPSWITCH environment variable in the root login shell if Commvault services are running as root user, and then restart Commvault services.
SAPSWITCH=/oracle/C19/newsapswicth
export SAPSWITCH=/oracle/C19/newsapswicth
When util_file_online device job runs in the SAP backup detail file, the environmental variable appears as shown below:
br_env SAPSWITCH=/oracle/C19/newsapswicth