VOPS0004: Backup or restore hangs during I/O on Ceph storage

Symptom

During a backup or restore operation, the I/O process hangs on Ceph cinder volumes.

Cause

The default soft limit of file descriptors (FDs) in Red Hat Enterprise Linux is 1024. If this setting is not overridden in the qemu.conf file, then the QEMU process uses this number.

If the QEMU process for the VSA proxy instance exceeds the soft limit on file descriptors while data is transferring from Ceph Block Devices (RBD), this can cause the instance to lock up.

To determine whether this is the issue, perform the following steps:

1. Run the following command to check the current soft limit on open files for the process:

   # cat /proc/$pid/limits | grep "Max open files"
   

   Where pid is the PID of the QEMU process for the VSA proxy instance where the I/O hang occurred.

2. Run the following command to determine the current number of opened file descriptors of the process:

   # echo /proc/$pid/fd/* | wc -w
   

This value should not be greater than the soft limit for file descriptors.

For more information, see Ceph - VM hangs when transferring large amounts of data to RBD disk.

Resolution

1. In the /etc/libvirt/qemu.conf file, increase the values for max_files and max_processes.

2. Restart libvirtd.