VOPS0001: Backup fails with error "Failed to detach disk"

Symptom

An OpenStack backup fails and leftover volume entries are not cleaned up. The following error is included in the vsbkp.log file:

 
Failed to detach disk [disk-guid], Error: [The resource could not be found.]

You can also see a message such as the following in the nova-api.log file:

Out of resources when opening file ... (Errcode: 24 "Too many open files")

Cause

This issue is caused by database exceptions that cause OpenStack services to become unresponsive. As a result, volumes cannot be detached from the proxy and leftover volume entries are not cleaned up.

Resolution

To resolve this issue, set the open_files_limit parameter in MySQL to 32768:

1. Check the current limit:

   [root@oprdo ~(keystone_admin)]# ps -ef| grep mysql
   root      51544  46858  0 23:40 pts/8    00:00:00 grep --color=auto mysql
   mysql    138296      1  2 Aug11 ?        04:25:22 /usr/libexec/mysqld --basedir=/usr
   [root@oprdo ~(keystone_admin)]# cat /proc/138296/limits | grep -i "open"
   Max open files            1024                 4096                 files
   [root@oprdo ~(keystone_admin)]# mysql -u root
   MariaDB [(none)]> SHOW VARIABLES LIKE 'open_files_limit';
   +------------------+-------+
   | Variable_name    | Value |
   +------------------+-------+
   | open_files_limit | 1024  |
   +------------------+-------+
   1 row in set (0.00 sec)
   

2. Increase the value for the open_files_limit parameter:

   1. Add the following content in the /etc/my.cnf configuration file:

      [mysqld]
      open_files_limit=32768
      

   2. In the systemd configuration file /etc/systemd/system/mariadb.service.d/limits.conf, increase the value for the LimitNOFILE setting:

      [Service]
      LimitNOFILE=32768
      

3. Restart services:

   $ systemctl daemon-reloa
   $ systemctl restart mariadb
   $ Openstack-service restart
   

4. Verify the limit:

   [root@oprdo ~(keystone_admin)]# !ps
   ps -ef| grep mysql
   mysql     63453      1  0 00:10 ?        00:00:06 /usr/libexec/mysqld --basedir=/usr
   root      68506  46858  0 00:22 pts/8    00:00:00 grep --color=auto mysql
   [root@oprdo ~(keystone_admin)]# cat /proc/63453/limits | grep -i "open"
   Max open files            32768                32768                files
   [root@oprdo ~(keystone_admin)]# mysql -u root
   MariaDB [(none)]> show global variables like 'open%';
   +------------------+-------+
   | Variable_name    | Value |
   +------------------+-------+
   | open_files_limit | 32768 |
   +------------------+-------+
   1 row in set (0.00 sec)