Troubleshooting Restore - Linux File System

The following sections provide information on some of the troubleshooting scenarios related to restore:

Browse and Restore

• Symptom

  Some of the files included in the user-defined subclient are not restored.

  Solution

  You may be performing the restore of user-defined subclient along with the restore of default subclient.

  When you are recovering data backed up by the default subclient, you cannot recover the data backed up by a user-defined subclient.

• Symptom

  Browse from default subclient displays No Backup at Time error.

  Solution

  Ensure that you do not include the root directory (/) as the subclient content.

  When performing point-in-time restore from the default subclient, include the data/folders under the root directory as the subclient content.

• Symptom

  Restore fails when trying to restore to a Unix FAT32 directory.

  Solution

  You may be restoring the full contents of a Unix directory that contains more than 32,767 files to a single Unix FAT32 directory.

  The number of entries in a single FAT32 directory cannot exceed 32,767.

• Symptom

  Restores from UNIX to Windows fails.

  Solution

  Restores from UNIX to Windows may fail in the following circumstances:

  If the files contain UNIX-specific device files such as block, character, or named pipe.

  If the UNIX files contain the "?" character in their filename.

  Windows allows 1024 characters for filenames, including the path. A filename, including the path, with more than 1024 (1023 for AIX) characters will not be restored from a UNIX computer to a Windows computer.

  If the UNIX files contain hard links.

  Make sure to avoid these while restoring the file system from UNIX to Windows.

• Case-Sensitive Filenames Will Not Restore from UNIX to Windows

  Case-sensitive filenames will not restore from UNIX to Windows because Windows does not maintain case-sensitive filenames. For example, if you are trying to restore files named "CASE_SENSITIVE" and "case_sensitive" from UNIX to Windows, then whichever file is restored last will overwrite the first one that was restored.

• Symptom

  ACLs and other extended attributes may not be restored.

  Solution

  Sometimes, when you restore data to an NFS-mounted file system, ACLs and other extended attributes may not be restored.

• Symptom

  Restore fails when trying to restore a running executable file.

  Solution

  Ensure that you are not including any running executable files in the restore operation.

• LNX0002: Example of Linux full system restore on an LVM based server

  This example will show an In-Place Restore of a complete Linux operating system that was installed using LVM based storage.

  Prior to performing a restore of this nature the existing operating system must be examined to capture file system sizes and disk partition layouts.

  This can be done using the following commands:

  cat /etc/fstab

  cat /proc/partitions

  fdisk -l

  df

  lvm

  1. pvdisplay sub command within lvm

  2. vgdisplay

  3. lvdisplay

  The following is the captured output from these commands run on a Linux computer that will be used to demonstrate this restore:

  # cat /etc/fstab

  /dev/VolGroup00/LogVol00 / ext3 defaults 1 1
  /dev/VolGroup00/LogVol03 /var ext3 defaults 1 2
  /dev/VolGroup00/LogVol02 /usr ext3 defaults 1 2
  LABEL=/boot /boot ext3 defaults 1 2
  tmpfs /dev/shm tmpfs defaults 0 0
  devpts /dev/pts devpts gid=5,mode=620 0 0
  sysfs /sys sysfs defaults 0 0
  proc /proc proc defaults 0 0
  /dev/VolGroup00/LogVol01 swap swap defaults 0 0

  # cat /proc/partitions

  major minor #blocks name
  8 0 10485760 sda
  8 1 104391 sda1
  8 2 10377990 sda2
  253 0 1572864 dm-0
  253 1 1441792 dm-1
  253 2 5242880 dm-2
  253 3 2097152 dm-3

  # fdisk -l

  Disk /dev/sda: 10.7 GB, 10737418240 bytes
  255 heads, 63 sectors/track, 1305 cylinders
  Units = cylinders of 16065 * 512 = 8225280 bytes
  Device Boot Start End Blocks Id System
  /dev/sda1 * 1 13 104391 83 Linux
  /dev/sda2 14 1305 10377990 8e Linux LVM

  # df

  Filesystem 1K-blocks Used Available Use% Mounted on
  /dev/mapper/VolGroup00-LogVol00 1523568 476028 968900 33% /
  /dev/mapper/VolGroup00-LogVol03 1396600 101824 1222688 8% /var
  /dev/mapper/VolGroup00-LogVol02 5078656 2892516 1923996 61% /usr
  /dev/sda1 101086 12634 83233 14% /boot
  tmpfs 1029784 0 1029784 0% /dev/shm

  # lvm
  lvm>
  lvm> pvdisplay

  --- Physical volume ---
  PV Name /dev/sda2
  VG Name VolGroup00
  PV Size 9.90 GB / not usable 22.76 MB
  Allocatable yes (but full)
  PE Size (KByte) 32768
  Total PE 316
  Free PE 0
  Allocated PE 316
  PV UUID BmTTrd-Tpe9-Lf6D-ic0f-HGCj-4E31-FrWJ7y

  lvm> vgdisplay

  --- Volume group ---
  VG Name VolGroup00
  System ID
  Format lvm2
  Metadata Areas 1
  Metadata Sequence No 5
  VG Access read/write
  VG Status resizable
  MAX LV 0
  Cur LV 4
  Open LV 4
  Max PV 0
  Cur PV 1
  Act PV 1
  VG Size 9.88 GB
  PE Size 32.00 MB
  Total PE 316
  Alloc PE / Size 316 / 9.88 GB
  Free PE / Size 0 / 0
  VG UUID x94Vk1-MeGP-TTGG-FRhb-jUdZ-WA7q-SujSiy

  lvm> lvdisplay

  --- Logical volume ---
  LV Name /dev/VolGroup00/LogVol00
  VG Name VolGroup00
  LV UUID XAFA3v-1D3k-S646-7Hzg-AC8c-2Xjt-mT6fW0
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 1.50 GB
  Current LE 48
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 253:0
  --- Logical volume ---
  LV Name /dev/VolGroup00/LogVol03
  VG Name VolGroup00
  LV UUID jDCqyo-0r0A-UbiM-wmZi-jdf7-4g3U-JPqiJN
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 1.38 GB
  Current LE 44
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 253:1
  --- Logical volume ---
  LV Name /dev/VolGroup00/LogVol02
  VG Name VolGroup00
  LV UUID F7ZuKi-Jn1C-sRxG-rw4F-unNy-o3ls-WE9r2Y
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 5.00 GB
  Current LE 160
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 253:2
  --- Logical volume ---
  LV Name /dev/VolGroup00/LogVol01
  VG Name VolGroup00
  LV UUID JtwYci-7RhO-JPvC-ZGSN-u1Zq-qeTz-c9KsCF
  LV Write Access read/write
  LV Status available
  # open 1
  LV Size 2.00 GB
  Current LE 64
  Segments 1
  Allocation inherit
  Read ahead sectors auto
  - currently set to 256
  Block device 253:3

  lvm> exit

Solution

Once this information has been gathered, and a full backup has been run on the Linux computer, the client needs to be powered off, and the system disk has to be replaced with a same sized disk. In addition to this disk, another disk also needs to be added to install a basic Linux operating system which will be used to restore the original disk. Be sure to replace the original system disk with the same type of disk using the same target ID. Add the additional new disk to a different SCSI ID from the original system disk. This will ensure that reconfiguring the restored operating system goes smoothly.

On this computer, the original 10GB sda disk is replaced with another sda disk (SCSI target 0), and a new 4GB sdb disk (SCSI target 1) is added. A bare minimum Linux operating system will be installed onto the sdb disk which will then get Commvault file system agent installed on to it. At that point a restore will be performed, restoring the data back on to sda (as it was originally installed).

From the output above it can be noted that the root (/) partition was located on /dev/VolGroup00/LogVol00, a swap partition was located on /dev/VolGroup00/LogVol01, /usr was located on /dev/VolGroup00/LogVol02, /var was located on /dev/VolGroup00/LogVol03, and the /boot partition was located on a regular non-LVM /dev/sda1 partition. The VolGroup00 was located on disk partition /dev/sda2.

Here is a table showing the original disk layout:

The disk layout on /dev/sda will be re-created during the installation of Linux operating system onto the second disk (sdb).

When installing Linux operating system on the second disk, the grub boot loader will get installed onto /dev/sda, and will point to /dev/sda1 for the boot information. A swap partition will be created on /dev/sda2 in LogVol1, which will be what is used for the Linux install to sdb, so no swap partition will be needed on /dev/sdb.

For the install to the sdb disk, the following partitions should be configured on /dev/sda and /dev/sdb during the installation. Make sure to size the partitions and logical volumes appropriately using the output gathered in the beginning of this example.

Below is a table showing the disk layout and mount points of sda & sdb prior to restore:

1. After the operating system is installed onto the sdb disk, renew the client certificates. See Renewing a Revoked Certificate for step-by-step instructions.

2. Install the File System iDataAgent (using the standard defaults).

3. Unmount the /dev/sda1 partition from /boot and re-mount it under the /mnt/restore/boot directory to restore the original boot files.

   For example:

   # mkdir /mnt/restore/boot

   # umount /boot

   # mount /dev/sda1 /mnt/restore/boot

4. Initiate a restore from the CommCell Console to restore / (the entire system) to the /mnt/restore directory. Select the UnconditionallOverwrite option to force the restore to overwrite the files in /mnt/restore/boot.

5. After the restore completes, a /proc directory needs to be created in the restored root partition. This can be completed using the command

   mkdir /mnt/restore/proc

6. Set up the boot loader to boot from the newly restored disk sda. The following procedure will accomplish this task..

   1. Mount proc & dev to the top level of the restored root file system

   2. Run the chroot command to point to the /restore partition/directory

   3. Run the grub install on the sda device.

   Here is an example execution of this procedure:

   # mount -t proc none /mnt/restore/proc

   # mount -o bind /dev /mnt/restore/dev

   # chroot /mnt/restore

   # grub-install /dev/sda
   Installation finished. No error reported.
   This is the contents of the device map /boot/grub/device.map.
   Check if this is correct or not. If any of the lines is incorrect,fix it and re-run the script `grub-install'.
   # this device map was generated by anaconda
   (fd0) /dev/fd0
   (hd0) /dev/sda

7. Copy the operating system files that have hardware information in them to the /mnt/restore directory if the hardware on the computer is different from that of the source computer.

   For example, on RHEL compatible operating systems like CentOS and Oracle Linux, you need to copy the following files:

   cp –rp /etc/udev/rules.d/70-persistent-net.rules /mnt/restore/etc/udev/rules.d/70-persistent-net.rules
   cp –rp /etc/sysconfig/network-scripts/ifcfg-ethX /mnt/restore/etc/sysconfig/network-scripts/ifcfg-ethX
   cp –rp /etc/fstab /mnt/restore/etc/fstab

   Remove the /mnt/restore/etc/blkid/blkid.tab file if present.

   For example, on RHEL compatible operating systems like CentOS and Oracle Linux, you need to exclude the following files:

8. Edit the /mnt/restore/etc/fstab file to remove /mnt/restore from the mount points:

   For example:

   /mnt/restore -> /

   /mnt/restore/usr -> /usr

   /mnt/restore/var -> var

9. At this point the computer can be powered off, and the sdb disk can be removed. After removing the second disk, the computer will boot normally using the newly restored LVM partitioned sda disk.

10. Renew the client certificates again. See Renewing a Revoked Certificate for step-by-step instructions.

Installation

• Symptom

  Error while loading shared libraries.

  Solution

  On Linux clients, the below error appears when we run any process or service:

  <process name>: error while loading shared libraries: <lib>.so: cannot enable executable stack as shared object requires: Permission denied

  For example,

  #./ifind_scan -getmnt -all
  ./ifind_scan: error while loading shared libraries: libCvOnTap.so: cannot enable executable stack as shared object requires: Permission denied

  As a workaround, do the following steps:

  1. Check if ASL (Atomic Secured Linux) is configured on client.

     #uname –r
     2.6.32.59-17.art.i686.PAE

     .art indicates that ASL is configured on the client.

  2. Check for the presence of the below logs in /var/log/messages file.

     May 24 22:01:08 rhel6 kernel: Aborting core
     May 24 22:01:08 rhel6 kernel: PAX: execution attempt in: <anonymous mapping>, bfc46000-bfc5b000 bffeb000
     May 24 22:01:08 rhel6 kernel: PAX: terminating task: /usr/libexec/paxtest/mprotstack(mprotstack):13201, uid/euid: 0/0, PC: bfc5acf4, SP: bfc5acdc
     May 24 22:01:08 rhel6 kernel: PAX: bytes at PC: c3 1a a3 ae 2b ac 9f ae f4 0f 9f ae 00 00 00 00 f4 0f 9f ae

  3. Run the following command.

      
     chpax –ps ../iDataAgent/_process/service

     For example:

     #chpax –ps /opt/commvault/iDataAgent/ifind_scan