Oracle IntelliSnap Troubleshooting

Updated

IntelliSnap Backup Failures

Failed in Indexing Operation (Create or Update Index)

Make sure to perform a crosscheck archive log all operation. Then, retry the IntelliSnap backup operation.

Failed during VSS IntelliSnap Backup

Verify the Windows Event Viewer, if any error occurs during the VSS IntelliSnap backup operation.

There is a known issue with Oracle VSS Writer memory leak (metalink ID 1358570.1). If Oravssw.exe uses more memory, restart the service.

IntelliSnap Backup Fails

IntelliSnap Backup operations fail if the database is in the NOARCHIVELOG mode. Alter the database to ARCHIVELOG mode and then perform the IntelliSnap backups.

SQL>startup mount;
        
    
SQL>alter database archivelog;
        
    
SQL>alter database open;
        
    

Sometimes, the IntelliSnap backup operations may fail if you enable the SSKIPBACKUPBROWSE additional setting. Disable it.

To add or edit an additional setting, follow the steps in Add or Modify an Additional Setting.

Mount Operation Fails on Windows

During RMAN backup copy operation on proxy, the mount operations fails with the following error message:

OS mount failed : [Mount failed: E_FAIL (MM.60611)]
        
    

To resolve this issue make sure the mount directory is empty and there are no drives (same as source data drives) exist on the proxy.

IntelliSnap Restores of High Transaction Rate Databases May Fail

Do not use IntelliSnap to back up archive log volume of high transaction databases, because the following errors may be encountered while restoring from those snaps or during the backup copy of those IntelliSnap jobs.

ORA-00283: recovery session canceled due to errors

ORA-00354: corrupt redo log block header

ORA-00353: log corruption near block 86016 change 7963079914 time 01/28/2014 09:54:46

Instead back up the archive logs using RMAN. Select the Use Rman for Log Backup check box on the Logs Backup tab of the Subclient Properties dialog box.

IntelliSnap Backup Job Runs Indefinitely

IntelliSnap Backup operations may run indefinitely due to following reasons:

  • If the archive log location is full. In such cases, you have to either clear the archive logs to make enough space available or specify a different archive log location.

  • If there are faulty or failed multipath devices. If faulty devices exist, delete them using the following command on a linux computer.

    dmsetup message <faulty multipath device> 0 "fail_if_no_path" 
     multipath -f <faulty multipath device>
            
        

Oracle RMAN Snap to Tape Incremental Copy Fails on the Proxy Computer

When performing an Oracle RMAN incremental snap to tape copy, note the following:

  1. The Oracle database installed on the proxy machine should be of the same version as the source. For example, if Oracle 10.2.0.4 is installed on source then the proxy also should be of the same version.

  2. Oracle user ID/group ID on source and proxy should be the same otherwise the RMAN backup copy will fail with permission issues.

  3. Copy the Oracle parameter file pfile from the source to the proxy (say the instance only as spfile)

    sqlplus <username/password@servicename> as sysdba << EOF 
     Create pfile from spfile;
     Exit; 
     EOF
            
        

    Copy the pfile init<instance name>.ora to the proxy computer and the destination location should be $ORACLE_HOME/dbs/ with oracle user permissions. Also, copy the oracle password file from the source to the proxy computer's $ORACLE_HOME/dbs/ directory.

  4. Create the bdump, udump, adump, cdump and diagnostic_dest directories. Please note that the directories should be in the same location as the source.

  5. Create the directories DB_CREATE_FILE_DEST, LOG_ARCHIVE_DEST and any other directory required for starting the database in NOMOUNT mode. If there are multiple archive destinations, then create the directories for each of the archivelog destinations.

  6. Copy $ORACLE_HOME/network/admin/tnsnames.ora configuration from source to proxy. If the entire content cannot be copied then copy at least the configuration related to catalog connection.

  7. Startup the proxy instance in NOMOUNT mode.

  8. Now configure the proxy Oracle Instance on the CommServe Console and status should be started. Now you are all set to do Oracle RMAN snap to tape incremental

Note that for incremental backup, snap clone will be mounted in the same location as the source mount-point of the source database. For example, if the data mount-point is /netapp/data then on the proxy too it will be mounted in /netapp/data, similarly for the archive log location chosen for the IntelliSnap backup. Therefore, ensure that on the proxy this mount-point is free and there is no such directory existing on the proxy computer ( even if it exists it should be empty).

IntelliSnap Backup Copy on the Proxy Fails

Sometimes, the Backup copy on proxy fails with the following error: ORA-7217 sltln: environment variable cannot be evaluated.

Example:

If RMAN configuration parameters contain $s as shown below:

RMAN> show all;

RMAN configuration parameters for database with db_unique_name are:

CONFIGURE RETENTION POLICY TO RECOVERY WINDOW OF 0 DAYS;
 CONFIGURE BACKUP OPTIMIZATION ON;
 CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
 CONFIGURE CONTROLFILE AUTOBACKUP OFF;
 CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK TO '$ORACLE_BKUP/$ORACLE_SID/controlfile%F.f';
 CONFIGURE DEVICE TYPE DISK PARALLELISM 2 BACKUP TYPE TO COMPRESSED BACKUPSET;
 CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
 CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1; # default
 CONFIGURE CHANNEL DEVICE TYPE DISK FORMAT '$ORACLE_BKUP/$ORACLE_SID/%d_df_%t_b%s_p%p.rmf' MAXPIECESIZE 2000 M;
 CONFIGURE MAXSETSIZE TO UNLIMITED; # default
 CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
 CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default
 CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE 'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
 CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default
 CONFIGURE SNAPSHOT CONTROLFILE NAME TO '$ORACLE_BKUP/$ORACLE_SID/snapcf.f';
 ORA-7217 sltln: environment variable cannot be evaluated
        
    

Perform one of the following steps to resolve this issue:

  • Configure “/” as connect string on proxy and set the environment variables specified for snapshot control file path in cvprofile residing in Base directory.

    1. Edit /opt/commvault/Base/cvprofile

    2. export ORACLE_BKUP=/tmp

    3. commvault restart

  • Change the configuration parameters on source to remove environment variables ($ORACLE_BKUP etc).

Sometimes, the backup copy on proxy fails with the following error even though the control file copy is cataloged into recovery catalog:

RMAN-03002: failure of backup command at 04/11/2012 13:44:03
 RMAN-06004: ORACLE error from recovery catalog database: RMAN-20220: control file copy not found in the recovery catalog
 RMAN-06090: error while looking up control file copy: +DATA/backup.ctl.galaxy
 RMAN>
        
    

Perform the following to resolve this failure:

  • Unregister and then reregister the source database to recovery catalog.

  • Resume the backup copy job.

Sometimes the IntelliSnap operation and a backup copy on proxy for ASM database fails with the following errors:

For Snap:

Error Code: [19:1335]

Description: Oracle Backup [GetASMLogDisks Failed.]

For Backup Copy:

Error Code: [19:1335]

Description: Oracle Backup [Mounting snap or renaming ASM DiskGroup operation failed with an error. Please check the logs for more details.]

30156 f48b7410 04/18 13:00:37 97550 OraObject::GetOraMode() - oraMode = SHUTDOWN.
 30156 f48b7410 04/18 13:00:37 97550 OraObject::GetOraMode() - oraMode = SHUTDOWN: return Error.
 30156 f48b7410 04/18 13:00:37 97550 OraInfoBase::GetInfo() - CheckOraMode() failed: oraError=301989906
 30156 f48b7410 04/18 13:00:37 97550 ASMSnapUtil::runSqlWithScript() - Failed while getting the Oracle version
 30156 f48b7410 04/18 13:00:37 97550 ASMSnapUtil::runSqlWithScript() - Writing into file [@/opt/commvault/Base/Temp/tmp_asm_30156.sql] sql = [select 'U,'|| state from v$asm_diskgroup where name = 'DATADG1'
 /
 ]
 30156 f48b7410 04/18 13:00:37 97550 ASMSnapUtil::runSqlWithScript() - Executing SQL select 'U,'|| state from v$asm_diskgroup where name = 'DATADG1'
 /
 failed with an error Database is in SHUTDOWN mode
 30156 f48b7410 04/18 13:00:37 97550 ASMSnapUtil::isASMDiskGroupMounted() - Failed while executing the sqlscript [select 'U,'|| state from v$asm_diskgroup where name = 'DATADG1'
 /
 ] output = []
 30156 f48b7410 04/18 13:00:37 97550 ASMDiskGroup::renameASMDiskGroup() - Child change user=oracle, gid=501, uid=501
 30156 f48b7410 04/18 13:00:37 97550 ClOraSnapAgent::RenameAndMountASMDiskGroup() - Successfully Renamed ASM DISK GROUPS
 30156 f48b7410 04/18 13:00:37 97550 OraObject::GetOraMode() - strictSID = 0
 30156 f48b7410 04/18 13:00:37 97550 OraChildProcess::SetPostForkParam() - Parent path = /oracle11gr2/product/11.2.0/dbhome_1/bin/sqlplus
 30156 f48b7410 04/18 13:00:37 97550 OraChildProcess::SetPostForkParam() - Parent oraUser = oracle
        
    

Perform the following to resolve this failure:

  1. Log in to the CommCell Console.

  2. Verify the Oracle +ASM instance status. The status should be in started mode.

  3. Resume the IntelliSnap operation or the backup copy job that you need to perform.

Latency Occurs for Releasing File Descriptors by an ASM Instance during the Dismount of ASM Diskgroups

On Oracle version 11.2.02, there is a latency for releasing file descriptors by ASM instance during dismounting of ASM diskgroups.

Apply the following patch on Oracle version 11.2.02 to resolve the latency issue:

Patch:11666137

https://updates.oracle.com/download/11666137.html

IntelliSnap Backup Copy of an Oracle ASM Database Fails If You Perform an IntelliSnap Operation without Disabling the Snap Integrity for Persistent Snap Engines

If you perform a IntelliSnap operation without disabling the Snap Integrity for persistent snap engines, the following error is displayed:

19935 b7f476d0 03/01 12:20:03 11972 ASMDiskGroup::mountASMDisk() - Failed while excuting sql [alter system set asm_diskstring='/ora_snap*/asm*','/ora_snap*/*'
 /
alter diskgroup DATA mount
/
] output = [ SQL*Plus: Release 11.2.0.1.0 Production on Thu Mar 1 12:20:03 2012,Copyright (c) 1982, 2009, Oracle. All rights reserved.,SQL> SQL> SQL> SQL> Connected.,SQL> SQL> SQL> SQL> SQL> SQL> SQL> ,System altered.,alter diskgroup DATA mount,ERROR at line 1:,ORA-15032: not all alterations performed,ORA-15017: diskgroup "DATA" cannot be mounted,ORA-15063: ASM discovered an insufficient number of disks for diskgroup "DATA",SQL> SQL> Disconnected from Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 - Production,With the Automatic Storage Management option,]
19935 b7f476d0 03/01 12:20:03 11972 ASMDiskGroup::mountASMDisk() - One of the diskgroups could not be mounted
 19935 b7f476d0 03/01 12:20:03 11972 ClOraSnapAgent::RenameAndMountASMDiskGroup() - mountASMDiskGroup failed
        
    

Perform the following steps to resolve this issue:

  1. Mount the snaps from the CommCell console (List Snapshots-->Mount) on Proxy and then use the following command:

    /opt/commvault/Base/diskgroup_rename.pl -y <original source diskgroup name> <mounted disk path>
            
        

    For example, if snap is mounted on /tmp and the original diskgroup name is DATA01, then perform the following command:

    [oracle@Hindu ~]$ /opt/commvault/Base/diskgroup_rename.pl -y DATA01 /tmp/volume_1335990104/DATA01
     renaming disks to: DATA01
     current name for /tmp/volume_1335990104/DATA01: DS2995770
    new name for /tmp/volume_1335990104/DATA01: DATA01
            
        
  2. Unmount the Snap. Perform the same for all the snaps in that IntelliSnap operation.

Changing the Snapshot control file and SP file Location to a Snappable Volume

Run the following RMAN command to change the path for the snapshot control file:

Configure Snapshot CONTROLFILE NAME TO '<Snappable volme>\SNCFVSSDB.ora';
        
    

Remove the SP file from the $ORACLE_HOME/database location or move sp file to snappable location and add this SPfile location to pfile.

Oracle VSS IntelliSnap Job Fails on Windows

XML Document is Long

Oracle VSS Snap job fails with "failed to finalize shadow" and windows events show the following error message:

XML document is too long.

Oracle stores the redo logs in BCD XML. Hence, the XML file size may be large if there are a lot of transactions. Therefore, rerun the IntelliSnap operation when there is less I/O activity on the oracle database.

Device Not Filer Volume

When performing VSS IntelliSnap job after restoring the control file, the job may fail with one of the following errors:

Device is not filer volume
 Device Not Found
        
    

This happens because the Oracle VSS writer includes a snapshot control file for RMAN, which by default is in the %ORACLE_HOME%/database location and may not be on a snapable volume.

To resolve this, run the following RMAN script to change the location of the snapshot control file:

RMAN> CONFIGURE SNAPSHOT CONTROLFILE NAME TO '<Snappable Volume>\SNAPSHOT.CTL';
        
    

Example:

RMAN> CONFIGURE SNAPSHOT CONTROLFILE NAME TO 'F:\SNAPDB_LOG\SNAPTEST\SNAPSHOT.CT L';
        
    

Output:

using target database control file instead of recovery catalog 
 old RMAN configuration parameters: CONFIGURE SNAPSHOT CONTROLFILE NAME TO 'F:\SNAPDB_LOG\SNAPTEST\SNAPSHOT.CTL'; 
 new RMAN configuration parameters: CONFIGURE SNAPSHOT CONTROLFILE NAME TO 'F:\SNAPDB_LOG\SNAPTEST\SNAPSHOT.CTL'; 
 new RMAN configuration parameters are successfully stored 
 RMAN> exit
        
    

Tablespace and Data file Restores from Snap That use the File System Fail on Windows

During partial tablespace/data file restore from snap using File System, if the database is open, the data file restore operation may fail with busy errors.

To resolve this issue, make sure the database is in mount mode before running partial restore from Snap using File System.

Oracle 11.2.0.3 and Higher Errors

When using Oracle 11.2.0.3 version or higher, if the database ID for auxiliary instance is changed using NID script, the table restore from snap or File System backup copy and database cloning restores will fail with the following error message:

NID-00106: LOGIN to target database failed with Oracle error:

ORA-01017: invalid username/password; logon denied

To resolve this issue, install the Oracle update 13366202 on the destination database.

SP File Creation Fails on the Proxy

Backup copy operations fail with the following error message on the proxy computer:

Failed to create spfile on proxy
        
    

To resolve this issue, ensure the following:

  • The listener should be configured and in running state

  • Pfile/spfile should not contain any invalid parameters

  • All directories mentioned in the pfile/spfile must be created correctly on the proxy computer.

Completed with One or More Errors

Backup jobs from Oracle iDataAgent will be displayed as "Completed w/ one or more errors" in the Job History in the following cases:

  • When RMAN Script execution for the backup job completes with warnings.

  • When job is killed after backing up some data.

During offline backups, if the database cannot be opened after a backup. The software reports "Completed w/ one or more errors" for restore jobs in the following cases:

  • During a table restore, if the export or import of table fails.

  • RMAN recovery is completed, but an incorrect open mode is selected for restore.

Catalog Errors during IntelliSnap Backup

During IntelliSnap backup job, you may notice the following catalog errors:

File Name: /opt/commvault/MediaAgent/SnapVolumeMounts/SnapMnt_1_2_26099/ oradata/ONLINE/dbconf.cfg RMAN-07517: Reason: The file header is corrupted

File Name: /opt/commvault/MediaAgent/SnapVolumeMounts/SnapMnt_1_2_26099/ oradata/ONLINE/initONLINE.ora RMAN-07517: Reason: The file header is corrupted

File Name: /opt/commvault/MediaAgent/SnapVolumeMounts/SnapMnt_1_2_26099/ oradata/ONLINE/spfileONLINE.ora RMAN-07518: Reason: Foreign database file DBID: 0 Database Name:

File Name: /opt/commvault/MediaAgent/SnapVolumeMounts/SnapMnt_1_2_26099/ oradata/ONLINE/GalaxyControlFile.Conf RMAN-07517: Reason: The file header is corrupted

These error messages can be ignored. As part of the backup job, files such as spfile, pfile and backup controlfiles are copied to the archive log location. Oracle does not recognize these files as archive log files and hence displays the error messages.

Recover until Time Fails because of a Corrupt Archive Redo Log

Issue

The snapping of the archive log volume may not be consistent if there are high transactions and frequent log switches happening.

Symptom

The Recover Until Time fails with the Oracle error ORA-00353: log corruption near block. This can happen when there are frequent online redo log switches.

Resolution

The subclient needs to have the Use Rman for Log Backup option enabled. Right-click the subclient and select Properties. On the Subclient Properties dialog box, navigate to the Logs Backup tab and select the Use Rman for Log Backup check box. Retry the operation.

ORCL0044: The Oracle Wallet May Not Open after an Offline Backup and Restore

Issue:

If the Oracle Wallet is defined in the default location (ORACLE_BASE) and there are no entries to the ORACLE_WALLET location in the sqlnet.ra file, the Oracle Wallet may not open after an offline backup and restore when either of the following is true:

  • The subclient is configured with no lights-out script (Subclient Properties dialog box, Advanced tab, Offline Arguments tab)

  • The Switch Database Mode for Restore option (Options tab) was used for the restore.

Solution

Use one of the following options to fix this issue.

  • Define the Oracle Wallet location in the sqlnet.ra file

  • Configure a post-script that opens the Oracle Wallet (Subclient Properties dialog box, Pre/Post Process tab, PostBackup Process box)

ORCL0045: An IntelliSnap Backup Copy Fails when the ASM_DISKSTRING Parameter Is Not Set

Issue

An IntelliSnap backup copy fails for an Oracle database that runs in an ASM configuration when the ASM_DISKSTRING parameter is not set.

The error is similar to the following

SQL>alter system set asm_diskstring='/opt/commvault/MediaAgent/SnapVolumeMounts/SnapMnt_1_*/*' * ERROR at line 1: ORA-02097: parameter cannot be modified because specified value is invalid ORA-15014: path '/dev/dm-19' is not in the discovery set

Solution

Set the Oracle ASM_DISKSTRING initialization parameter to a value that allows ASM to discover all of the ASM disks, for example, /dev/dm-*.

ORCL0046: Some Channels Unexpectedly Terminate during a Multistream IntelliSnap Oracle 12c Backup Copy

Issue

Some channels unexpectedly terminate during a multistream IntelliSnap Oracle 12c backup copy before the operation completes.

Solution

Check the value of the PGA_AGGREGATE_LIMIT database parameter and increase it. The minimum default value that Oracle recommends is 2 GB. You can get more information in Oracle Doc ID 1520324.1.

Backup using NetApp cluster mode LUNs fails with error: "Snapshot operation not allowed due to clones backed by snapshots"

For more information, see KB article 84806.