Loading...

HyperScale Reference Architecture - Building Resiliency

Architectural Resiliency

Platform resilience is a function of system architecture and best practices implemented to deliver the required level of service. On Commvault's HyperScale platform, the inherent application level resilience of a distributed deduplication database (DDB) and Index, is complimented by the scale-out architecture which uses standard servers with redundant components. Implementing industry best-practices such as mirrored root disk, RAID for cache SSD's, hot spares, separate VLAN's for public data protection traffic and private storage pool traffic over bonded network interfaces, further enhances resilience at the node-level.

Disk and node level resilience on the Commvault HyperScale platform is a function of erasure coding scheme and the block-size of the cluster. Erasure code determines number of chunks the incoming data is broken into before being saved to HDD's (bricks), which are dispersed across nodes in a block. The block size is the number of server nodes in a block. The set of disks in all nodes within a block, used to house the dispersed chunks constitute a sub-volume. Higher tolerance to node level failures is achieved by dispersing the chunks across more number of nodes within the cluster – larger block-size. Higher tolerance against disk failures is achieved by selecting an appropriate erasure coding scheme – e.g:-(8+4) coding allows for up to four concurrent disk failures within each sub-volume while (4+2) only allows for up to two concurrent disk failures within a sub-volume. Following is an illustration of (4+2) erasure code, on a block of three (3) nodes.

resiliency_01

In the above example with erasure code of (4+2), the sub-volume size is 6xHDD's. This cluster can tolerate the loss of any two HDD's for each sub-volume in the cluster or the failure of a complete node. This is because both these failure conditions will result in the availability of the minimum number of data blocks - four (4), for continued operation. Adding more sub-volumes through the addition of more HDD's will not alter tolerance against node failures as all sub-volumes will be impacted when a node fails.

Following is a table of resiliency for different erasure codes and block-sizes. The erasure code scheme to use and block-size of the cluster have a bearing on both resilience and disk capacity growth needs of a customer.

Erasure Code

Block Size

(Nodes / Block)

Sub- Volume Size (HDD's

Erasure Code Disk Overhead

Tolerance to Node Failures

Tolerance to HDD Failures

Comments

(4 + 2)

3 Nodes

6

33%

1 node per block

2 HDD's per sub-volume

Basic configuration. More usable capacity than 2-way replication

6 Nodes

6

33%

2 nodes per block

2 HDD's per sub-volume

Good balance of resiliency with disk capacity and scaling in-place. Resiliency equivalent to 3-way replication with more usable disk capacity.

(8 + 4)

3 Nodes

12

33%

1 node per block

4 HDD's per sub-volume

Better resiliency against disk failures. Starter for future cluster growth.

6 Nodes

12

33%

2 nodes per block

4 HDD's per sub-volume

Better overall resiliency and disk capacity scaling options.

12 Nodes

12

33%

4 nodes per block

4 HDD's per sub-volume

Fully scaled configuration with maximum resiliency.

HyperScale installer provides for separation of the operating system (OS) from the Index cache/deduplication database (DDB) and data tiers. This three-tiered deployment model allows for the OS to be installed on mirrored disks, when available, while the SSD tier can be used for Index cache and DDB layers. Network resilience is provided through interface bonding. The default mode adopted is “Round-Robin”(mode 0). The mode can be easily changed, if required, to be compatible with any environment. It is important to ensure the selected bonding mode is supported by the HyperScale software, underlying Operating System (OS) and customer network. Bonding mode has a bearing on both network resilience and available bandwidth.

Network resilience at the server level is provided either through interface bonding or vendor recommended methods. If interface bonding is used, It is important to ensure the bonding mode is supported by the HyperScale software, underlying Operating System (OS), server platform and customer network. Currently, Hyperscale modes supported are 0 (Round-Robin), 1 (Active-Standby) and 4 (LACP). Please check with the hardware vendor of your choice to determine the bonding modes supported on the specific server.

Resilience Tests

Resilience testing confirms the system can recover from component failures without the loss of data or functionality. On Commvault's HyperScale platform, a series of tests simulating component failures were introduced to ascertain functionality. In each instance, a long running backup job was monitored for change in status, if any. Tests conducted included failing disks, node and network components on both control and data nodes. Disk related failures covered root, data and cache SSD's. Where the disks were protected by RAID, the node with the component failure continued to be available. This is true of mirrored root disks and cache SSD's in a RAID setup. Since the capacity data disks are protected by (4+2) erasure code, loss of up to two HDD's per sub-volume does not impact the node. Similarly, the loss of an entire node within a block of three nodes does not have any impact on the data management function of the platform. This is true whether the affected node type is control or data. Network bonding, when implemented across 2x10G interfaces for each VLAN, contains redundancy across ports and north-bound switch connections. Network bonding mode determines the resilience and bandwidth potential of the bonded interfaces. In tests, the mode was set to “Active-Standby” (mode=1). When implemented as such, loss of a single port, connection or transceiver does not result in a network disconnect. However, when both interfaces in a bond are down, the particular VLAN is affected causing either the data protection (public) or the storage pool (private) network to become unavailable. This situation results in the node not able to provide data-protection services, until connectivity to both VLAN’s is restored. However, backup services are not affected as the other two nodes in the block are available. The loss of all network ports appears as a node-down to the CommServe and other nodes in the cluster. Given that a basic (4+2) erasure coding scheme allows for a node to be down, application functionality is not affected in this case as well.

In conclusion, application availability is within expected parameters for disk, node and network failures. Node level resilience is a factor of erasure coding for capacity disks (HDD's) and RAID for root and cache SSD's, number of bonded interfaces and bonding mode. The HyperScale block level resilience from node failures is also determined by the erasure coding scheme employed across all nodes in the block.

Related Topics

Resiliency

Last modified: 11/1/2019 6:08:49 PM