Troubleshooting Backups for Informix

INF0002: Command Line Backup Failure when a 64-bit Application is running on a 64-bit Solaris Client

Issue:

If a 64-bit application is running on a 64-bit Solaris client, the command line backup fails with the following error:

GetProcAddress failed, geterror=ld.so.1: onbar_d: fatal: invalid handle

Example:

bash-3.00$ onbar -b -L 0
 onbar is starting ..........
 Galaxy Instance: Instance001
 Loading Library:/opt/commvault/Base64/libCvIbsad.so from dir /opt/commvault/Base64
 LoadLibrary failed when loading libCvIbsad.so, error = ld.so.1: onbar_d: fatal: libCstd.so.1: open failed: No such file or directory
 onbar is starting ..........
 Galaxy Instance: Instance001
 Loading Library:/opt/commvault/Base64/libCvIbsad.so from dir /opt/commvault/Base64
 GetProcAddress failed, geterror=ld.so.1: onbar_d: fatal: invalid handle

Resolution:
Export LD_LIBRARY_PATH=<CommvaultInstall>/Base64 and run the command line jobs.

INF0003: Timeout Failures

Issue:

Backups fail due to timeouts.

Resolution:

The default time for resources to allocate streams during an Informix third-party command line backup job is 86400 seconds (i.e., 24 hours). If a backup fails due to a timeout (86400 seconds) being reached, you can configure the sALLOCATESTREAMSECS additional setting to increase the waiting time period.

INF0004: Onbar error code - 43207

Issue:

Case 1:

Backup jobs go to pending with following error in bar_act.log.

Windows:

2013-12-23 09:33:00 5468 5468 E:\Informix\12.10\bin\onbar_d.exe -b -L 0
 2013-12-23 09:33:00 5468 5468 Working with cvsm as generic storage manager.
 2013-12-23 09:33:00 5468 5468 Successfully connected to Storage Manager.
 2013-12-23 09:33:00 5468 5468 (-43207) Unable to open connection to database server: could not fork server connection, SQLCODE -25553.

Unix:

2014-02-24 11:10:10 10775 10769 onbar_d -b -L 0
 2014-02-24 11:10:10 10775 10769 Working with cvsm as generic storage manager.
 2014-02-24 11:10:10 10775 10769 Successfully connected to Storage Manager.
 2014-02-24 11:10:10 10775 10769 (-43207) Unable to open connection to database server: could not fork server connection, SQLCODE -25553.

Resolution

Check if the name of file containing sqlhosts information for the server is ‘sqlhosts’. If the name is different, add the InformixServername_SQLHOSTFile Additional Setting specifying full path to the sqlhosts file.

Use the onstat -g dis command to display the location of sqlhosts file for a particular server.

onstat -g dis command output:

There are 2 servers found
 Server : ol_informix1210_1
 Server Number : 111
 Server Type : IDS
 Server Status : Up
 Server Version: IBM Informix Dynamic Server Version 12.10.FC3
 Shared Memory : 0xa000000 
INFORMIXDIR : /INFORMIX/ol_informix1210_1 
ONCONFIG : /INFORMIX/ol_informix1210_1/etc/onconfig.ol_informix1210_1 
SQLHOSTS : /INFORMIX/ol_informix1210_1/etc/sqlhosts.ol_informix1210_1 
Host : breezer 
Server : ol_informix1210_2 
Server Number : 222 
Server Type : IDS 
Server Status : Down
 Server Version: IBM Informix Dynamic Server Version 12.10.FC1 
Shared Memory : 0xa000000 
INFORMIXDIR : /INFORMIX/ol_informix1210_2 
ONCONFIG : /INFORMIX/ol_informix1210_2/etc/onconfig.ol_informix1210_2 
SQLHOSTS : /INFORMIX/ol_informix1210_2/etc/sqlhosts.ol_informix1210_2 
Host : breezer

Add the following additional settings to protect these two servers.

ol_informix1210_1_SQLHOSTFile /INFORMIX/ol_informix1210_1/etc/sqlhosts.ol_informix1210_1

ol_informix1210_2_SQLHOSTFile /INFORMIX/ol_informix1210_1/etc/sqlhosts.ol_informix1210_2

Case 2

Backup jobs go to pending with following error in bar_act.log.

2014-02-24 11:12:27 10974 10967 onbar_d -b -L 0
 2014-02-24 11:12:27 10974 10967 Working with cvsm as generic storage manager.
 2014-02-24 11:12:27 10974 10967 Successfully connected to Storage Manager
 2014-02-24 11:12:27 10974 10967 (-43207) Unable to open connection to database server: could not fork server connection, SQLCODE -25596.

Resolution:

Check if the entries in sqlhosts file are correct.

INF0005: Command Line Backup Fails when a Log Backup Is Triggered by the Alarm Program Interferes with a Full Backup

Issue

A command line backup fails with the following error message:

2015-09-28 00:10:43 3863 3861 /informix/bin/onbar_d -b -l 
 2015-09-28 00:10:44 3863 3861 Working with cvsm as generic storage manager. 
 2015-09-28 00:10:44 3863 3861 (-43395) A log backup is already running. Can't start another. 
 2015-09-28 00:10:49 3863 3861 /informix/bin/onbar_d complete, returning 152 (0x98)

Resolution

This occurs when you have set the BAR_MAX_BACKUP onconfig parameter is set to 1.

1. Increase the number of streams by setting the BAR_MAX_BACKUP onconfig parameter.

2. If there is no log space restriction, use the log_full.sh script in place of the alarmprogram.sh script to trigger the backups for full logical logs.

This delays the start of each log backup and prevents it from being triggered for each individual log.

Best Practice: Copy and save the log_full.sh script file before you modify the file.

Add the following lines to the log_full.sh script file.

Set the parameters as:

 
################################ 
 # Added the following to only backup logs 
 # when there are N or more logs to be backed up. 
 echo "$3" >> $INFORMIXDIR/Log_Cnt 
 COUNT=`cat $INFORMIXDIR/Log_Cnt |wc -l` 
 if [ $COUNT -ge N ] 
 then 
     export CvClientName= <client_name_as_seen_in_commcell_console> 
     export CvInstanceName=<Instance00x> 
     $BACKUP_CMD 2>&1 > /dev/null 
     #Clear Log_Cnt after backing up logs... 
     cat /dev/null > $INFORMIXDIR/Log_Cnt 
 fi 
 ################################

Example

To trigger the log backup after there are 5 or more full logical logs, set the parameters as:

################################ 
 # Added the following to only backup logs 
 # when there are 5 or more logs to be backed up. 
 echo "$3" >> $INFORMIXDIR/Log_Cnt 
 COUNT=`cat $INFORMIXDIR/Log_Cnt |wc -l` 
 if [ $COUNT -ge 5 ] 
 then 
     export CvClientName= informix_client 
     export CvInstanceName=Instance001 
     $BACKUP_CMD 2>&1 > /dev/null 
     #Clear Log_Cnt after backing up logs... 
     cat /dev/null > $INFORMIXDIR/Log_Cnt 
 fi 
 ################################

INF0006: A Command Line Job Fails to Load the Library on UNIX Clients

Issue:

A backup or restore command line job fails to load the library with the following error.

bash-3.2$ onbar -b -L 0 
 onbar is starting .......... 
 Galaxy Instance: Instance003 
 Loading Library:/opt/Commvault/Base32/libCvIbsad.so from dir /opt/Commvault/Base32 
 LoadLibrary failed when loading libCvIbsad.so, error = ld.so.1: onbar_d: fatal: /opt/Commvault/Base/libCvLib.so: wrong ELF class: ELFCLASS64onbar is starting .......... 
 Galaxy Instance: Instance003 
 Loading Library: /opt/Commvault/Base32/libCvIbsad.so from dir /opt/commvault/Base32 
 GetProcAddress failed, geterror=ld.so.1: onbar_d: fatal: invalid handle: 0x0onbar is starting .......... 
 Galaxy Instance: Instance003 
 Loading Library:/opt/Commvault/Base32/libCvIbsad.so from dir /opt/Commvault/Base32 
 GetProcAddress failed, geterror=ld.so.1: onbar_d: fatal: invalid handle: 0x0

Resolution:

Relink the Informix instance.

INF0007: A Reinstalled Client Can Cause the Backup to Go to the Pending State

Issue

If you reinstall a client to a different location, or manually install the client to a different location, the backup can go to the pending state. The log displays the following:

UNIX

This is the bar_act.log file.

2015-12-10 11:43:02 15601 15595 onbar_d -b -L 0 
 2015-12-10 11:43:02 15601 15595 (-43231) An unexpected error occurred: /opt/simpana/Base/ibsad001.so: cannot open shared object file: No such file or directory . 
 2015-12-10 11:43:02 15601 15595 (-43259) Could not open XBSA library /opt/simpana/Base/ibsad001.so, so trying default path.

Windows

This is the Bar_act.log file.

2016-03-07 15:36:13 11040 11040 E:\IFXSER~1\12.10\bin\onbar_d -b -L 0 
 2016-03-07 15:36:16 11040 11040 (-43075) Unable to open file C:\PROGRA~1\COMMVA~1\Simpana\Base\IFXXBSA.dll . 
 2016-03-07 15:36:16 11040 11040 Windows Error 127: The specified procedure could not be found.

Cause

The software cannot find the Informix library. This is the ibsad001.so library on UNIX and the IFXXBSA.dll library on Windows.

Resolution

UNIX Configuration

1. You must run the ifx_install.sh script to update the symlink path. For more information on this procedure, see Relink the Informix instance.

2. Update the BAR_BSALIB_PATH entry in the onconfig file.

Windows Configuration

1. Update the BAR_BSALIB_PATH entry in the onconfig file.

2. If you run INFORMIX 11.50 or 11.70, restart the Informix server.

Completed with One or More Errors

Backup jobs from Informix iDataAgent will be displayed as "Completed w/ one or more errors" in the Job History in the following cases:

• Job is killed after creating some valid data archive files. For example: after a database backup is run, if job is killed during the Logs phase.