Backup and restore operations fail
If backups and restore operations fail or if there are any issues with Commvault services, check if any denial messages are logged by SELinux. These denials are also known as AVC denials, and are logged to a different location, depending on which daemons are running.
Daemon |
Log location |
---|---|
auditd on |
/var/log/audit/audit.log |
auditd off; rsyslogd on |
/var/log/messages |
setroubleshootd, rsyslogd, and auditd on |
/var/log/audit/audit.log Easy-to-read denial messages are also sent to /var/log/messages file. |
Procedure
-
Go to /var/log/audit directory.
-
Open the audit.log file.
-
Check if any permissions are denied by SELinux.
A denial message appears as follows:
type=AVC msg=audit(1494408684.449:244): avc: denied { write } for pid=12037 comm="cvd" name="cvlaunchd.sock" dev=dm-0 ino=2110807 scontext=staff_u:sysadm_r:sysadm_t:s0-s15:c0.c1023 tcontext=staff_u:object_r:usr_t:s0 tclass=sock_file type=AVC msg=audit(1494408688.061:258): avc: denied { ioctl } for pid=12383 comm="CvMountd" path="/dev/sg0" dev=devtmpfs ino=10311 scontext=staff_u:sysadm_r:sysadm_t:s0-s15:c0.c1023 tcontext=system_u:object_r:scsi_generic_device_t:s0 tclass=chr_file
-
To give proper permissions, complete the following steps:
-
Log on to the client computer as a root user.
-
Run the following command if denial is from a single process command. For example, cvd.
grep "cvd" /var/log/audit/audit.log | audit2allow -M backup_IDA_1
-
Run the following command if denial is from multiple process commands. For example, cvd, Cvmountd, and so on.
grep -E "cvd|Cvmountd" /var/log/audit/audit.log | audit2allow -M backup_IDA_1
-
Run the command to install the policy module.
semodule -i backup_IDA1.pp
-
Perform backup and restore operations to see if any other permissions are denied. If yes, go back to step #4 and repeat the procedure.
-
MediaAgent package is installed earlier and operations such as browse and synthetic full backup fail
If the MediaAgent package is installed earlier and operations such as browse and synthetic full backup fail, then configure the following settings:
Procedure
-
Verify that the client has sufficient memory. A minimum of 8 GB is recommended.
-
Increase the number of system file descriptor limit. You can modify the number of concurrently open file descriptors throughout the system using the /etc/sysctl.conf file under Linux operating system.
-
Log on to the client computer as a root user.
-
Increase the maximum number of open files by setting a new value in the kernel variable /proc/sys/fs/file-max.
# sysctl -w fs.file-max=100000
The command forces the limit to 100000 files.
-
Edit the /etc/sysctl.conf file and insert the following line so that after a reboot the setting persists.
# vi /etc/sysctl.conf
-
Append a configuration directive.
fs.file-max = 100000
-
Save and close the /etc/sysctl.conf file.
Log out and log back on to the client computer for changes to take effect or run the following command:
# sysctl -p
-
Verify your settings.
# cat /proc/sys/fs/file-max
-
-
Increase the number of user level file descriptor limit.
-
You can limit specific users to specific file descriptor limits by editing the /etc/security/limits.conf file.
# vi /etc/security/limits.conf
-
Set user name to soft, and the set the hard limits.
username soft nofile 4096 username hard nofile 10240
-
Save and close the /etc/security/limits.conf file. To see the configured limits, use the following commands.
# su - username $ ulimit -Hn $ ulimit -Sn
-
-
Edit the /etc/pam.d/login file and add or modify the following line.
session required pam_limits.so
-
Save and close the /etc/pam.d/login file.
-
Restart Commvault services.
commvault restart