Loading...

Data Pipe

Table of Contents

Overview

Data Pipe Buffers

Network Agents

Application Read Size

Overview

The Data Pipe is the logical mechanism through which data is transferred between a client and a MediaAgent. You configure and optimize the Data Pipe using the Data Pipe Buffers, Network Agents and Network Bandwidth Throttling, and Application Read Size parameters. All these parameter settings should be optimized, so that data transfer is not only fast and lossless, but also does not use more resources than necessary.

Data Pipe Buffers

The Data Pipe Buffers setting controls the amount of shared memory allocated for data pipes on each client or MediaAgent computer. By default, the system allocates 30 pipeline buffers of 64KB size on each client computer. (30 buffers times 64KB each allocates about 2 MB of shared memory.)

You can configure additional buffers for transferring data between the client and the MediaAgent. Having more pipeline buffers may translate into faster dedicated (non-multiplexed) backups to tape devices, and therefore improve data transfer performance. Of course, additional buffers consume more shared memory.

Configuring Data Pipe Buffers

Caution: This is an advanced feature. Please use caution when modifying this setting.

Important: You can set the number of data pipe buffers on the MediaAgent or the client. If the value is set both on the client and the MediaAgent, the setting on the client is used.

Change the default data pipe buffers value (30) as follows:

  • Setting the number of pipeline buffers and the pipeline buffer size from the subclient level from the Commcell Console. For more information, see subclient specific performance tab.
  • The number of pipeline buffers can also be set by using the nNumPipelineBuffers additional settings. For more information on configuring an additional settings see, Add or Modify an Additional Setting.

Network Agents

Network Agents establish the data pipes. By default, the system uses two concurrent network agents, which may be suitable for 100-megabit networks. Increasing this value may increase the data transfer throughput from the client. Other network features, such as NIC speed and network switch speed, also impact network throughput. You may need to experiment in your environment to determine whether you can realize faster data throughput by changing the number of concurrent network agents.

Setting the Network Bandwidth and Network Agents for a Data Protection Operation

Note: All subclients support network bandwidth throttling, except subclients that use NAS agents.

To set a limit for network bandwidth and the number of concurrent network agents for a subclient backup operation, perform the following steps:

  1. In the CommCell Browser, expand Client Computers > client_name > agent > backup set or instance.

    Note: If limits are set at the instance level and the subclient level, the subclient settings override the instance settings.

  2. Right-click the subclient and then click Properties.

    The Subclient Properties dialog box appears.

  3. For file system subclients, update the following:
    1. In the Subclient Properties dialog box, click Advanced.

      The Advanced Subclient Properties dialog box appears.

    2. On the Performance tab, in the Network Agents box, set the maximum number of network agents to run concurrently when performing a subclient backup.
    3. Select the Throttle Network Bandwidth (MB/HR) check box and then enter the throughput limit. The Commvault software throttles the transfer on a per-job basis.
    4. Click OK.
  4. For non-file system subclients, on the Storage Device tab, click the Data Transfer Option tab and then update the following:
    1. In the Network Agents box, set the maximum number of network agents to run concurrently when performing a subclient backup.
    2. Select the Throttle Network Bandwidth (MB/HR) check box and then enter the throughput limit. The Commvault software throttles the transfer on a per-job basis.
  5. Click OK.

Setting the Network Bandwidth and Network Agents for Multiple Subclients Using Subclient Policy

For agents that support subclient policies, network agents can be established for creating multiple subclients using an existing subclient policy.

Use the following steps to set the network bandwidth and network agents while creating multiple subclients:

  1. From the CommCell Browser, expand Policies.
  2. Right-click Subclient Policies, and then click New Subclient Policy.
  3. In the Name box, enter a name for the subclient policy.
  4. Click the iDataAgent list and then select Windows File System.
  5. Click once in the Storage Policy column cell and then select a storage policy for the default subclient.
  6. Click the Association tab, and then select the backup set on the client that you want to associate with this subclient policy.
  7. Click the General tab, select the default subclient row, and then click Edit.

    The Create New Subclient dialog box appears.

  8. On the Content tab, enter the contents for the subclient by browsing to them or by entering their path. If browsing, select a file or folder as content and then click Add.
  9. Click Close when you are done adding content.
  10. On the Performance tab, perform the following steps:
    1. In the Network Agents box, enter the number of network agents that must be used to perform backup operations on the subclient.
    2. Select the Throttle Network Bandwidth (MB/HR) check box and enter the throughput limit. The Commvault software throttles the transfer on a per-network-agent basis.
  11. Click OK.

The default subclient is automatically created in all the clients with which the subclient policy is associated.

Application Read Size

Application read size is the size of the application data read from the clients for data transfer during backup operations. Values for Application Read Size must be in the power of 2; minimum value is 64 KB, maximum value is 4096 KB (4 MB).

Note that each application internally allocates a buffer size that is suitable for handling the application data. When the size of the application data read during backup operations matches the source application's internal buffer allocation then the overhead is minimized. So to achieve optimal rate of data transfer during backups, configure this value based on the source application's buffer allocation.

The value of application read size may be increased to reduce the amount of data read from the given application, which in turn reduces the amount of I/O operations performed against it. As a result, overall backup performance may increase. However, backup memory usage may also increase, which may inadvertently consume additional resources from the application. Thus, it is recommended that this value be set at either the default value or match the block size directed by the application.

This configuration is supported for the following agents:

  • Microsoft Windows File System Agent
  • Microsoft SQL Server Agent

Modifying Application Read Size

Use these steps to set the application read size:

  1. In the CommCell Browser, expand Client Computers > client >agent > backup set or instance.
  2. Right-click the subclient, and then click Properties.

    The Subclient Properties dialog box is displayed.

  3. For Windows file system subclients, do the following:
    1. Click Advanced.

      The Advanced Subclient Properties dialog box is displayed.

    2. On the Performance tab, select Application Read Size and select the number of kilobytes.
  4. For Microsoft SQL Server subclients, do the following:
    1. On the Storage Device tab, click the Data Transfer Option tab.
    2. Select Application Read Size and select the number of kilobytes.
  5. Click OK.

Last modified: 1/23/2018 6:10:54 PM