Upgrading the Hosted Engine From Version 4.1 to 4.3

The virtual machine (VM) containing the Hosted Engine (or the oVirt engine) can be upgraded from version 4.1 to 4.3.

The upgrade is a 3 step process as follows:

• First, the hosted engine must be upgraded to the latest 4.1 minor version.

• Next, the hosted engine must be upgraded to 4.2.

• Finally, the hosted engine must be upgraded to 4.3.

Before You Begin

• Ensure that the MediaAgent nodes are upgraded to the latest kernel.

  • For more information about upgrading the operating system, refer to Installing Operating System Updates, if the MediaAgent is upgraded to Service Pack version 15 (or above).

  • If the MediaAgent is in Service Pack version 14 (or below), manually upgrade the repositories to the latest kernel.

• Obtain the oVirt admin login and password. You will need the credentials to perform the upgrade.

Procedure

Pre-Upgrade TasksMediaAgent

1. Contact Commvault customer support and obtain an USB drive containing the RPMs (Red Hat Package Manager) needed for the upgrade.

2. Copy the contents of the USB drive to /ws/ddb/rpms.

3. Make /ws/ddb/rpms as a NFS share, so that it can be accessed from the hosted-engine VM.

4. From the MediaAgent node set the hosted engine in the maintenance mode.

   hosted-engine --set-maintenance --mode=global
   

5. Login to the hosted engine VM as root.

6. Create a directory /repos and mount the directory:

   For example:

   mount MA_1:/ws/ddb/rpms /repos
   

7. Unpack the CVAppliance_Oct2019.repo.tar and copy CVAppliance_Oct2019.repo to /etc/yum.repos.d on the Hosted Engine.

8. Unpack the CVAppliance_Oct2019.tar and copy the contents of the HyperVHost/CVAppliance/Unix/rhel-7.7 folder to /ws/ddb/rpms on the MediaAgent from which RHV upgrade is planned.

Upgrade to the Latest 4.1 Minor Version

1. Update your manager (or RHEL host hypervisor) so that ansible is at version 2.9.13. You can do this by disabling the package that is too new by adding the following line to /etc/yum.conf file.

   Raw
   exclude=ansible-2.9.1[4-9] ansible-2.9.[2-9][0-9]*
   

2. Upgrade to the latest 4.1 minor version using the following commands:

   yum-config-manager --disable rhel-7-server-rhv-4.2-manager-rpms
   yum-config-manager --disable rhel-7-server-rhv-4.3-manager-rpms
   engine-upgrade-check
   

   Make sure that the result of the above command comes out successfully with “upgrade is available” message.

   engine-upgrade-check 
   yum update ovirt\*setup\*
   engine-setup
   yum update
   

   The following prompts will be displayed:

   1. Would you like to proceed? (Yes, No) [Yes]:

      Press <Enter> to continue.

   2. PACKAGE: [update] vdsm-jsonrpc-java-1.3.16-1.el7ev.noarch``do you wish to update them now? (Yes, No) [Yes]:

      Press <Enter> to continue.

   3. Setup can automatically configure the firewall on this system.``Note: automatic configuration of the firewall may overwrite current settings.``Do you want Setup to configure the firewall? (Yes, No) [Yes]:

      Type No and then press <Enter> to continue.

   4. Would you like to backup the existing database before upgrading it? (Yes, No) [Yes]:

      Press <Enter> to continue.

   5. Perform full vacuum on the engine database engine@localhost?``This operation may take a while depending on this setup health and the configuration of the db vacuum process.``See https://www.postgresql.org/docs/9.0/static/sql-vacuum.html``(Yes, No) [No]:

      Press <Enter> to continue.

   6. Please confirm installation settings (OK, Cancel) [OK]:

      Press <Enter> to continue.

   Continue the upgrade (to the latest 4.1 minor version) using the following command:

   yum update
   

   You may see transaction check errors similar to the following:

   file /opt/rh/eap7/root/usr/share/java/weld-spi-sources.jar from install of eap7-jboss-weld-3.0-api-weld-spi-0:3.0.0-7.SP4_redhat_00001.1.el7eap.noarch conflicts with file from package eap7-jboss-weld-2.2-api-0:2.3.0-7.SP1_redhat_1.1.ep7.el7.noarch
   file /opt/rh/eap7/root/usr/share/java/weld-spi.jar from install of eap7-jboss-weld-3.0-api-weld-spi-0:3.0.0-7.SP4_redhat_00001.1.el7eap.noarch conflicts with file from package eap7-jboss-weld-2.2-api-0:2.3.0-7.SP1_redhat_1.1.ep7.el7.noarch
   file /opt/rh/eap7/root/usr/share/java/weld-api-sources.jar from install of eap7-jboss-weld-3.0-api-weld-api-0:3.0.0-7.SP4_redhat_00001.1.el7eap.noarch conflicts with file from package eap7-jboss-weld-2.2-api-0:2.3.0-7.SP1_redhat_1.1.ep7.el7.noarch
   file /opt/rh/eap7/root/usr/share/java/weld-api.jar from install of eap7-jboss-weld-3.0-api-weld-api-0:3.0.0-7.SP4_redhat_00001.1.el7eap.noarch conflicts with file from package eap7-jboss-weld-2.2-api-0:2.3.0-7.SP1_redhat_1.1.ep7.el7.noarch
   

3. If you see these errors, run the following commands to resolve these errors. (Replace the file names with the file names displayed as errors when you run the command.)

   rpm -Uvh --force /repos/jb-eap-7-for-rhel-7-server-rpms/Packages/e/eap7-jboss-weld-3.0-api-weld-spi-3.0.0-7.SP4_redhat_00001.1.el7eap.noarch.rpm
   rpm --force -Uvh /repos/jb-eap-7-for-rhel-7-server-rpms/Packages/e/eap7-jboss-weld-3.0-api-weld-api-3.0.0-7.SP4_redhat_00001.1.el7eap.noarch.rpm
   

   Re-run the following command:

   yum update
   

4. Login to one of the MediaAgent nodes and determine the node in which the hosted engine is running using the following command:

   hosted-engine –vm-status
   

5. Login to node in which the hosted engine VM is running and restart the VM using the following commands:

   hosted-engine --vm-shutdown
   hosted-engine --vm-poweroff
   hosted-engine --vm-start
   

   The hosted engine VM kernel will be upgraded to the latest kernel version available in the payload. (The hosted engine will continue to be in the maintenance mode.) Use the following command to verify the kernel version:

   uname –a
   

6. Login to the hosted engine VM as root.

7. Mount the directory/repository containing the packages.

   For example:

   mount MA_1:/ws/ddb/rpms /repos
   

Upgrade to Version 4.2

1. Upgrade to version 4.2 using the following commands:

   yum-config-manager --enable rhel-7-server-rhv-4.2-manager-rpms
   engine-upgrade-check
   

   Make sure that the result of the above command comes out successfully with “upgrade is available” message.

   engine-upgrade-check 
   yum update ovirt\*setup\*
   engine-setup
   

   The following prompts will be displayed:

   1. Would you like to proceed? (Yes, No) [Yes]:

      Press <Enter> to continue.

   2. --== PRODUCT OPTIONS ==-``Configure ovirt-provider-ovn (Yes, No) [Yes]:

      Press <Enter> to continue.

   3. --== PACKAGES ==--``Replying "No" will abort Setup. You can pass the option "--offline" to prevent installing or updating packages.``Do you wish to update them now? (Yes, No) [Yes]:

      Press <Enter> to continue.

   4. --== NETWORK CONFIGURATION ==--``Setup can automatically configure the firewall on this system.``Note: automatic configuration of the firewall may overwrite current settings.``NOTICE: iptables is deprecated and will be removed in future releases``Do you want Setup to configure the firewall? (Yes, No) [Yes]:

      Type No and then press <Enter> to continue.

   5. --== DATABASE CONFIGURATION ==--``postgresql.conf is usually in /var/lib/pgsql/data, /var/opt/rh/rh-postgresql95/lib/pgsql/data, or somewhere under /etc/postgresql* . You have to restart PostgreSQL after making these changes.``The database requires these configurations values to be changed. Setup can fix them for you or abort. Fix automatically? (Yes, No) [Yes]:

      Press <Enter> to continue.

   6. [WARNING] This release requires PostgreSQL server 9.5.14 but the engine database is currently hosted on PostgreSQL server 9.2.24.``[ INFO ] Verifying PostgreSQL SELinux file context rules``This tool can automatically upgrade PostgreSQL. Automatically upgrade? (Yes, No) [Yes]:

      Press <Enter> to continue.

   7. PostgreSQL will be upgraded by copying its data to a new directory.``Do you want to automatically clean up the old data directory on success to reclaim its space (141 MB)? (Yes, No) [Yes]:

      Press <Enter> to continue.

   8. [ INFO ] Any further action on the DB will be performed only after PostgreSQL has been successfully upgraded to 9.5.``[ INFO ] Engine DB and DWH one shares the same PostgreSQL instance that is going to be upgraded``Perform full vacuum on the oVirt engine history database ovirt_engine_history@localhost?``This operation may take a while depending on this setup health and the configuration of the db vacuum process.``See https://www.postgresql.org/docs/9.0/static/sql-vacuum.html``(Yes, No) [No]:

      Press <Enter> to continue.

   9. --== OVIRT ENGINE CONFIGURATION ==--``Perform full vacuum on the engine database engine@localhost?``This operation may take a while depending on this setup health and the configuration of the db vacuum process.``See https://www.postgresql.org/docs/9.0/static/sql-vacuum.html``(Yes, No) [No]:

      Press <Enter> to continue.

   10. oVirt OVN provider user[admin@internal]:

       Type the oVirt admin name and press <Enter> to continue.

   11. oVirt OVN provider password:

       Type the password and press <Enter> to continue.

   12. [ INFO ] Stage: Setup validation``During execution engine service will be stopped (OK, Cancel) [OK]:

       Press <Enter> to continue.

   13. Please confirm installation settings (OK, Cancel) [OK]:

       Press <Enter> to continue.

       The following message should be displayed in the end:

       [ INFO ] Stage: Clean up``Log file is located at /var/log/ovirt-engine/setup/ovirt-engine-setup-20181214145558-0fsnjh.log``[ INFO ] Generating answer file '/var/lib/ovirt-engine/setup/answers/20181214150451-setup.conf'``[ INFO ] Stage: Pre-termination``[ INFO ] Stage: Termination``[ INFO ] Execution of setup completed successfully

2. Continue the upgrade (to 4.2 version) using the following commands:

   yum-config-manager --disable rhel-7-server-rhv-4.1-rpms
   yum-config-manager --disable rhel-7-server-rhv-4.1-manager-rpms
   yum-config-manager --disable rhel-7-server-rhv-4-tools-rpms
   yum-config-manager --disable jb-eap-7.1-for-rhel-7-server-rpms
   yum-config-manager --disable rhel-7-server-rhv-4.0-manager-rpms
   rpm -e --nodeps ovirt-imageio-common-1.5.1-0.el7ev.x86_64
   yum update
   

Upgrade to Version 4.3

1. Upgrade to version 4.3 using the following commands:

   yum-config-manager  --enable rhel-7-server-rhv-4.3-manager-rpms
   yum-config-manager  --enable jb-eap-7.2-for-rhel-7-server-rpms
   

   Make sure that the result of the above command comes out successfully with upgrade is available message.

   engine-upgrade-check 
   yum update ovirt\*setup\*
   engine-setup
   

   The following prompts will be displayed:

   1. Would you like to proceed? (Yes, No) [Yes]:

      Press <Enter> to continue.

   2. Set up Cinderlib integration (Currently in tech preview) (Yes, No) [No]:

      Press <Enter> to continue.

   3. Replying "No" will abort Setup. You can pass the option "--offline" to prevent installing or updating packages.``Do you wish to update them now? (Yes, No) [Yes]:

      Press <Enter> to continue.

   4. NOTICE: iptables is deprecated and will be removed in future releases``Do you want Setup to configure the firewall? (Yes, No) [Yes]:

      Type No and then press <Enter> to continue.

   5. Verifying PostgreSQL SELinux file context rules``This tool can automatically upgrade PostgreSQL. Automatically upgrade? (Yes, No) [Yes]:

      Press <Enter> to continue.

   6. PostgreSQL will be upgraded by copying its data to a new directory. Do you want to automatically clean up the old data directory on success to reclaim its space (147 MB)? (Yes, No) [Yes]:

      Press <Enter> to continue.

   7. Configuration of the db vacuum process.``See https://www.postgresql.org/docs/10/sql-vacuum.html``(Yes, No) [No]:

      Press <Enter> to continue.

   8. PostgreSQL will be upgraded by copying its data to a new directory.``Do you want to automatically clean up the old data directory on success to reclaim its space (141 MB)? (Yes, No) [Yes]:

      Press <Enter> to continue.

   9. --== OVIRT ENGINE CONFIGURATION ==--``Perform full vacuum on the engine database engine@localhost?``This operation may take a while depending on this setup health and the configuration of the db vacuum process.``See https://www.postgresql.org/docs/9.0/static/sql-vacuum.html``(Yes, No) [No]:

      Press <Enter> to continue.

   10. Setup validation``During execution engine service will be stopped (OK, Cancel) [OK]:

       Press <Enter> to continue.

   11. Configure VMConsole Proxy : True``Please confirm installation settings (OK, Cancel) [OK]:

       Press <Enter> to continue.

       This starts the upgrade and should complete with the following

       Execution of setup completed successfully.

2. Continue the upgrade (to 4.3 version) using the following commands:

    
   yum-config-manager --disable rhel-7-server-rhv-4.2-manager-rpms``yum-config-manager --disable jb-eap-7-for-rhel-7-server-rpms``yum update

Post-Upgrade Tasks

1. Change the compatibility version from 4.1 to 4.3 in the Administration Portal.

   Note

   A power cycle of the VM’s is needed for this to take effect. Hence this may require a scheduled down time for the VM hosting the CommServe.

   1. From the Administration Portal, click Compute > Clusters.

   2. Select the cluster to change and then click Edit.

   3. Change the Compatibility Version to the 4.3.

   4. Click OK twice to confirm the change.

      

2. Stop the VM hosting the CommServe.

   1. From the Administration Portal, click Compute > Virtual Machine.

   2. Right click the CommServe VM then click Shutdown.

      

3. Restart the VM hosting the CommServe.

   1. From the Administration Portal, click Compute > Virtual Machine.

   2. Right click the CommServe VM then click Run.

4. Remove the VM Lease.

   1. From the Administration Portal, click Compute > Virtual Machine.

   2. Right click the CommServe VM then click Edit.

   3. Change the Target Storage Domain for VM Lease to No VM Lease.

   4. Wait for completion pop-up on upper right corner.

      

5. Re-add the VM Lease.

   1. From the Administration Portal, click Compute > Virtual Machine.

   2. Right click the CommServe VM then click Edit.

   3. Change the Target Storage Domain for VM Lease to <VM_Name>. (For example, cvhcisd in the above image.)

   4. Wait for completion pop-up on upper right corner.

6. Stop the VM hosting the CommServe.

   1. From the Administration Portal, click Compute > Virtual Machine.

   2. Right click the CommServe VM then click Shutdown.

7. Restart the VM hosting the CommServe.

   1. From the Administration Portal, click Compute > Virtual Machine.

   2. Right click the CommServe VM then click Run.

8. Login to the MediaAgent node and remove the maintenance mode on the hosted engine.

   hosted- engine --set-maintenance --mode=none
   

9. Once the upgrade is completed, remove the repo folder and the tar files created in step (2) above from the DDB drive.

10. Run a DDB backup and make sure that it completes successfully.

    If it fails to create a snapshot, then edit /etc/lvm/lvm.conf to increase the value of thin_pool_autoextend_threshold to 80.

Additional Information

• https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/upgrade_guide/upgrading_from_4-1

• https://access.redhat.com/documentation/en-us/red_hat_virtualization/4.3/html/upgrade_guide/upgrading_from_4-2