Content, Filters, and Regular Expressions - Backup

Overview

Data protection content is defined at the subclient level for File System iDataAgents. The default subclient, created during installation, has the unique characteristic of including all protected data not explicitly covered by other subclients within the backup set. Additional subclients can be created by the administrator to separate and manage a subset of the backup set data.

For File System Agents, if you have defined overlapping content across multiple subclients within the backup set, the content is backed up by one subclient. Other subclients within the backup set skip the content.

In the case of application subclients, the inclusive feature of the default subclient means it can automatically "discover" all data requiring its protection. In some iDataAgents, this automatic discovery feature can be disabled, filtered (for example: Lotus Notes), or otherwise configured to assign content to other subclients (for example: Exchange mailboxes).

File System Content

In File System iDataAgents, the default subclient content is defined as "\" or "/" which is the symbol for the root of all file systems. In this case, "all file systems" means:

• All local file systems excluding those file systems on read-only devices such as CD, DVD or tape drives.

• Windows mounted file systems are included by default, but can be excluded by job in the Advanced Backup Options by clearing the Follow Mount Points option.

• NFS-mounted file systems are excluded by default, but can be included by specifically listing the mount points in the subclient content.

• Raw character or block devices are not included by default but can be included in the subclient content by explicitly adding the device name.

• Network shares can be explicitly included by adding their UNC data paths and a user/password with read access authority.

• Data in Remote Storage (that is, data managed by Microsoft's Remote Storage Service) can be included by selecting the backup option to do so.

In most cases, manually editing the content of the default subclient will disable its inclusive discover feature. Adding user-defined content to a subclient should be done in subclients other than the default subclient.

Adding System Folders for Windows File System to a Subclient

The system folders such as My Documents, Desktop, Music etc. are created by default for each user. If you want to include all such folders in a subclient, add following paths as a subclient content:

You can also use a Qcommand to create the subclient and add the content to it. For more information, refer to Command Line documentation.

Using Wildcards to Define Content

Regular expressions (or wildcard characters) can be used to define content in a subclient. Wildcard expressions are characters such as * or ?. Regular expressions include patterns such as [a-f] or *.[l-n]df. The terms are interchangeable.

You can use the regular expressions for defining content at any level in the data path. For example: F:\Users\[A-L]* or *.pst.

Wildcard Behavior on Newly Installed UNIX Clients

On newly installed UNIX clients, a wildcard path with a single trailing "*" backs up only the files and not the subdirectories.

For example, if you have the following content to back up:

# ls -R /wctest1
 /wctest1:
 dir1 f1 f2 f3
 /wctest1/dir1:
 f1 f2 f3

If you define the subclient content as follows:

Content:

/wctest1/*

The following content is backed up by clients from previous versions and clients upgraded from previous versions:

/wctest1/f1 /wctest1/dir1/f1 /wctest1/dir1/f2 /wctest1/dir1/f3 /wctest1/dir1/ /wctest1/f2 /wctest1/f3

The following content is backed up by newly installed clients:

/wctest1/f1
/wctest1/dir1/
/wctest1/f2
/wctest1/f3
/wctest1/

To be able to back up the subdirectories, you must modify the subclient content using one of the following ways:

• Content:

  /wctest1

• Content:

  /wctest1/**

Wildcard Behavior on Windows Clients

On Windows clients,

• If you define the filter as

  C:\test\data\**

  then only data folder is backed up and not the files inside data

• If you define the filter as

  C:\test\data\***

  or

  C:\test\data

  then the data folder is ignored in this case.

'***' indicates 'at any level'.

For example,

to backup all files with has a predefined wildcard '*xyz*' in them, the wild card expression to use as content would be

C:\***\*xyz*

To include the folders use the following expression:

C:\***\*xyz*\***

Defining Filters

Filtering unnecessary data from data protection operations can reduce backup/migration time, storage space, and recovery time. Most, but not all i DataAgents, include some filtering capability at the subclient level. Subclient filters for File System Agents can be defined for a path, directory, or file level.

Some application iDataAgents such as Exchange Mailbox, Lotus Notes Database, and Lotus Notes Document also allow filters to be defined.

There are four basic types of filters that can be used:

Exclusion Filters

Exclusion filters can be defined at the subclient level and exclude data from being unnecessarily protected.

1. From the CommCell Browser, expand Client Computers > client > File System > backup_set.

2. Right-click the backup_set, point to All Tasks, and then click New Subclient.

   The Subclient Properties dialog box appears.

3. Click the Filters tab.

4. In the Exclude these files/folder/patterns section, click Add.

   The Enter Path dialog box appears.

5. Enter the paths to exclude and then click OK.

   Tip: Each directory must be on a separate line.

6. Click OK to close the Enter Path dialog box.

7. Click OK to close the Subclient Properties dialog box.

Exception Filters

Exception filters can only be defined at the subclient level for supported agents. An Exception filter allows you to define directory or file exceptions to a filter defined in the exclusion section. For example, you can "exclude" the C:\ProgramData with the "exception" of C:\ProgramData\Documents directory.

1. From the CommCell Browser, expand Client Computers > client > File System > backup_set.

2. Right-click the backup_set, point to All Tasks, and then click New Subclient.

   The Subclient Properties dialog box appears.

3. Click the Filters tab.

4. In the Except these files/folder/patterns section, click Add.

   The Enter Path dialog box appears.

5. Enter the paths to exclude and then click OK.

   Tip: Each directory must be on a separate line.

6. Click OK to close the Enter Path dialog box.

7. Click OK to close the Subclient Properties dialog box.

Subclient Policy Filters

Subclient Policy filters are made up of Exclusion and Exception filters, and applied by a Subclient Policy. You can choose to not include Subclient Policy filters in a subclient and thereby define your own Exclusion and Exception filters for each subclient. See Filtering Rules for an overview.

Global Filters

Global Filters are made up of Exclusion filters that can be defined for each operating system such as Windows, Unix, and for the Exchange application. Both Linux and Macintosh operating systems fall under the Unix global filter. Filters defined at global level can be included or excluded in a subclient filter definition.

1. From the CommCell Browser, expand Client Computers > client > File System > backup_set.

2. Right-click the backup_set, point to All Tasks, and then click New Subclient.

   The Subclient Properties dialog box appears.

3. Click the Filters tab and clear the Include Policy Filters check box.

4. From the Include Global Filters list, select ON.

   This will allow filters set up under Control Panel (Global Filters) to be used for excluding data from backups for the selected subclient.

   Note

   These filters are used in addition to the filters set at the subclient level.

5. Click OK.

See Global Filters - Online Help for an overview.

Adding All Filters to Global Filters as a Group

The CommCell Console does not support the group addition of Global Filters. However, you can use Microsoft SQL Server to add them by editing the CommServ.dbo.GXGlobalParam table directly.

Direct modification of the metadata is not supported. Before you attempt any modification of the CommServ database, perform a Disaster Recovery backup and ensure that it can be recovered successfully.

The CommServ.dbo.GXGlobalParam table has two fields: name and value.

The Windows global filters are defined by the name "Windows FS Exclude Filters". The name is not present unless a global filter entry has been previously created.

Before continuing, add a global filter entry and click OK. The procedure below will overwrite the entry.

To add a group of global filters you can run an SQL statement for example:

UPDATE GXGlobalParam set value=<filter list> WHERE name='Windows FS Exclude Filters'.

The <filter list> is a string of filter entries separated by a space for example:

C:\Temp D:\Tmp E:\Junk.

If the filter entry itself contains a space as

C:\Program Files\*

then those spaces are replaced by "+1" for example:

C:\Program+1Files\*

Use the following steps to construct a string of filters:

1. Copy the recommended filter list below into Microsoft Word or enter your own set of filters. Each filter should be plain text on a separate line with no following spaces.

2. Use Find and Replace to look for spaces in the filter text and replace them with "+1". Clean up any extra +1's at the end of each line.

3. Use Find and Replace to look for the special character "Manual Line Break" and replace it with a space.

4. Insert "UPDATE GXGlobalParam SET value = '" at the beginning of the text string and "' WHERE name = 'Windows FS Exclude Filters'" at the end.

   Save the document as an unformatted text document.

5. Open SQL's Query Analyzer and set it to use the CommServ database. Open the file created in step 4 and execute it.

6. Open the Global Filters dialog in the CommCell Console and verify the filters you added.

Here's a sample of the SQL script:

UPDATE GXGlobalParam

set value = '*:\**\ntuser.dat* *:\**\usrclass.dat* *:\**\temp *:\**\tmp

*:\**\temporary+1internet+1files *:\**\*cache

*:\**\documents+1and+1settings\**\recent

*:\**\documents+1and+1settings\**\history *:\**\drvspace.* *:\**\recycled

*:\**\recycler *:\system+1volume+1information

*:\windows\downloaded+1program+1files *:\win*\hwinfo.dat *:\win*\prefetch

*:\win*\system.da0 *:\win*\user.da0 *:\win*\history *:\win*\historique

*:\win*\$NtServicePackUninstall$ *:\win*\$NtUninstall* *:\win*\applog *:\win*\inf

*:\win*\installer *:\win*\pchealth *:\win*\servicePackFiles *:\win*\sysbckup

*:\win*\**\spool\**\*.spl *:\**\ws_ftp.log *:\**\appdata\**\microsoft\**\recent

*:\**\application+1data\microsoft\**\recent *:\**\recent\*.lnk

*:\**\windows+1update.log *:\**\windows+1update+1setup+1files

*:\program+1files\**\uninstall+1information

*:\program+1files\windowsupdate'WHERE name = 'Windows FS Exclude Filters'

UNIX Filters

Wildcard behavior on UNIX clients:

• On clients from previous versions and clients upgraded from previous versions, a wildcard path with a single trailing "*" filters both the files and the subdirectories.

• On newly installed clients, a wildcard path with a single trailing "*" filters only the files and not the subdirectories.

• Filter strings are case-sensitive.

Example 1Content:

/dir

/dir1

/dir2

Exclude=*.*

Exception=*.pst

Only files ending with ".pst" will be backed up (under ALL directories)
in /dir, /dir1 and /dir2.

Example 2Content:

/dir

/dir1

/dir2

Exclude=*.*

Exception=cl*
Only files beginning with "cl" will be backed up (under ALL directories) in /dir, /dir1 and /dir2.

Example 3
Content:

/dir

/dir1

/dir2

Exclude=/dir1/**

Exception=cl*
All files under /dir will be backed up.
Only files beginning with "cl" under /dir1 will be backed up (under ALL directories).
All files under /dir2 will be backed up.

Example 4Content:

/dir

/dir1

/dir2

Exclude=/dir*/**

Exception=/dir1/dir*/**/cl*
All file names beginning with "cl" under any directory whose name begins with "dir" under /dir1 will be backed up.

Example 5

On newly installed UNIX clients, a wildcard path with a single trailing "*" filters only the files and not the subdirectories.

For example, if you have the following content to back up:

# ls -R /wctest1
 /wctest1:
 dir1 f1 f2 f3
 /wctest1/dir1:
 f1 f2 f3

If you define the subclient content as:

Content:

/wctest1

Exclude=/wctest1/*

The following content is filtered by clients from previous versions and clients upgraded from previous versions:

/wctest1/f1
/wctest1/dir1/f1
/wctest1/dir1/f2
/wctest1/dir1/f3
/wctest1/dir1/
/wctest1/f2
/wctest1/f3

The following content is filtered by :

/wctest1/f1
/wctest1/f2
/wctest1/f3
/wctest1/

To be able to filter the subdirectories, you must modify the filters using one of the following ways:

• Content:

  /wctest1

  Exclude=/wctest1

• Content:

  /wctest1

  Exclude=/wctest1/**

Editing Filters

Once data protection filters have been defined, entries can be added or deleted from the Subclient Properties (Filters) tab. In addition, some agents provide an editing capability that allows modification to existing filters without having to delete and re-add them.

Delete Exclusion Filter

1. From the CommCell Browser, expand Client Computers > client > File System > backup_set.

2. Right-click the backup_set, point to All Tasks, and then click New Subclient.

   The Subclient Properties dialog box appears.

3. Click the Filters tab.

4. Click Delete on the right of Exclude these files/folder/patterns.

   This will delete the path set in both Exclude these files/folder/patterns and Except for these files/folders.

5. Click OK.

Delete an Exception Filter

1. From the CommCell Browser, expand Client Computers > client > File System > backup_set.

2. Right-click the backup_set, point to All Tasks, and then click New Subclient.

   The Subclient Properties dialog box appears.

3. Click the Filters tab.

4. Click Delete on the right of Exclude these files/folder/patterns.

   This will delete the path set in Except for these files/folders. However, the exclusions will still remain valid.

5. Click OK.

Using Wildcards to Define Filters

The use of wildcards or regular expressions in defining filters is supported for most agents, but support for specific wildcards varies.

For more information, see the following:

• Wildcards for Exchange Agents

• Wildcards for Windows File System Subclients

• Wildcards for UNIX File System Subclients

• Wildcards for Find Data (Name/Pattern) - Lotus Notes Document

Exception filters that include wildcards and that point to folders or files below an excluded folder are not supported.

Vendors of NAS file systems implement filtering differently. For example, BlueArc and Network Appliance systems support only the "*" character either before or after the character string (for example, *tmp or cache*).

Inclusions, Exclusions, and Exceptions to Exclusions

Subclient content serves as an include filter that allows you to specify the portions of client data that you want included in data protection operations. The Exclude Filter is designed to tell the system which subset of the specified content should not be included in data protection operations. The following example illustrates these concepts:

Include= /space
Exclude= /space/temp

The above example shows that the volume called "space" will be included in backups for the subclient, excluding the "temp" directory. Suppose there were a subdirectory of "temp" called "keep" that you wanted to include, you could set up an Exception to the Exclusion Filter, as shown in the next example:

Include= /space
Exclude= /space/temp
Except= /space/temp/keep

This would result in the volume "space" being included in backups, excluding the "temp" directory, except for the "temp/keep" subdirectory.

All removable media drives including Drive A and B for both Unix and Windows should be excluded since the operating system might consider them to be floppy or disk drives.

Exceptions to exclusions can be thought of as patterns that should overlap (or match the same paths) as exclusions. This is useful for protecting data that would otherwise be filtered out by the Exclude filter. The Exception filter is not intended for use as a stand-alone include filter--the objects in the Exception filter must be a subset of objects in the Exclusion Filter.

Regular expressions (or wildcards) offer additional flexibility in defining the content to be included, excluded and/or excepted. The next example illustrates how wildcards may be used in filtering:

Include= /space
Exclude= /space/tem*/**
Except= /space/temp/keep

This example defines the content to be included as the "space" volume, excluding all subdirectories of "space" that begin with the letters "tem", except for the "temp/keep" subdirectory. Using wildcards in this way would exclude directories such as "space/temp", "space/templates", and "space/temporary_internet_files" from data protection operations. The Exception filter provides an exception to the exclusion of all subdirectories beginning with "tem", by allowing the "temp/keep" directory to be protected.

A more advanced example follows:

Include= C:\ and D:\ and E:\
Exclude= *:\**\temp*\**
Except=
C:\templates\

Drives C, D and E are the content to be included, excluding all directories that begin with "temp" from any level on any of those drives, except for the "templates" directory on drive C. This is just one example of the many possible combinations of wildcards that may be used in filtering.

Keep in mind the following rules with regard to filtering:

• If the path matches the exclusion AND it does not match any exception, it is excluded (recursively if a directory). Wildcard exceptions are NOT matched against the file system to add content to the subclient. For example, exclude "/space/temp" (no exceptions) means do not back up "/space/temp" (recursively if it is a directory).

• If the path matches an exclusion AND it matches an exception to exclusions, it is NOT excluded. An exclusion of "/space/tem*" does not match any of the same paths as an exception "/space/temp/keep", so the exception has no effect on what is backed up.

• For Windows File System iDataAgents, when change journal is used to perform the scan, the same filter cannot be used to filter both a file and a directory at the same time. For example, C:\temp\pattern*\** would filter out all directories that match this pattern at a certain directory level, and C:\temp\pattern* would filter out all files that match this pattern at a certain directory level. Using classic file scan instead of change journal would allow you to use the same filter for both file and directory objects that match the same pattern.

• For Unix File System iDataAgents, the same filter can be used to filter both files and directories at the same time. For example, "/etc/*.log" is valid for Unix to exclude both a file and a directory called "user.log".

• We recommend avoiding the filtering of subclient content in such a way as to limit it to only a given file type, such as *.pst or *.doc files. This sort of configuration would prevent other data from being protected on that volume. Because content is mutually exclusive across subclients, no other subclient would be able to back up the excluded data--which could result in data loss.

• Wildcards may be used only in the last level of the content path when specifying subclient content, provided that the Treat Characters as Regular Expressions option is enabled.

• On newly installed UNIX file system clients, a wildcard path with a single trailing “*” allows exception to exclusion of only the files and not the subdirectories.

Testing the Phase of Backup Job Without Running a Backup

Testing the scan phase allows you to determine which files/objects would be backed up, the number of objects, time taken for scan, etc., based on your backup options and filters.

1. Configure the subclient content and filters as desired, in a test backup set.

2. Run a full backup in such a way that it will fail after the scan phase has completed. One way to do this is to add a post-scan or pre-backup script, on the Subclient Properties (Pre/Post Process) tab, that will exit with an error (e.g., fail_post_scan.bat with the contents exit 1).

3. The pending job information in the Job Controller window will provide the path of the collect file which contains a list of objects scanned for backup.

4. Navigate to the collect file on the client to see the list of files and directories that would have been backed up.

5. After viewing the scan results, kill the suspended job from the Job Controller by right-clicking the job and selecting Kill. Click Yes to confirm.

6. Make any changes as needed to the content and filters configuration to achieve the desired set of files, and re-run the process.

7. Once you are satisfied with the configuration, you can apply these changes to a subclient that is used in production and remove the post-scan or pre-backup script.