Log Monitoring Using OpenSearch and Fluent Bit

You can analyze large amounts of log data from multiple clients in your environment by monitoring Commvault logs using OpenSearch and Fluent Bit. Fluent Bit collects logs from the clients and client groups and then sends the generated logs to OpenSearch. OpenSearch then displays the logs on its dashboard.

Before You Begin

• Identify the clients and client groups you want to monitor the logs for.

• Install and configure OpenSearch.

Step 1: Download and Execute the Fluent-bitInstallWorkflow Workflow

1. Download the Fluent-bitInstallWorkflow workflow from the Commvault store.

2. Deploy the Fluent-bitInstallWorkflow workflow

3. Execute the Fluent-bitInstallWorkflow workflow.
   Enter the clients and client groups as inputs for which Fluent Bit collects logs.

Step 2: Configure an Index Template and an Index State Management ( ISM ) Policy in OpenSearch

• In OpenSearch, go to Dev Tools and perform the following steps:

  1. Enter the following API to create an index template and define an index pattern: 'cvlogs_index_*'. Any index with a name that follows this pattern, for example, cvlogs_index_1, will follow the policies defined in the index template. You can modify this index pattern if you want to create an index with a different name.

     "PUT /_index_template/cvltlogs-index-template"{
     "index_patterns":[
         "cvlogs_index_*"
     ],
     "data_stream":{
     
     },
     "template":{
         "settings":{
             "index":{
                 "number_of_shards":3,
                 "number_of_replicas":0
             }
         },
         "mappings":{
             "properties":{
                 "@timestamp":{
                 "type":"date"
                 },
                 "Client":{
                 "type":"keyword",
                 "ignore_above":256
                 },
                 "Commserve":{
                 "type":"keyword",
                 "ignore_above":256
                 },
                 "LogInfo":{
                 "type":"text",
                 "analyzer":"simple"
                 },
                 "Pid":{
                 "type":"long",
                 "fields":{
                     "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                     }
                 }
                 },
                 "Tid":{
                 "type":"keyword",
                 "ignore_above":256
                 },
                 "containerId":{
                 "type":"keyword",
                 "ignore_above":256
                 },
                 "filename":{
                 "type":"keyword",
                 "ignore_above":256
                 },
                 "jobid":{
                 "type":"long",
                 "fields":{
                     "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                     }
                 }
                 },
                 "message":{
                 "type":"text",
                 "analyzer":"simple",
                 "fields":{
                     "keyword":{
                     "type":"keyword",
                     "ignore_above":256
                     }
                 }
                 },
                 "spanId":{
                 "type":"keyword",
                 "ignore_above":256
                 },
                 "traceId":{
                 "type":"keyword",
                 "ignore_above":256
                 }
             }
         }
     }
     }

  2. Enter the following API to create an Index State Management ( ISM ) policy that applies to all indices following the cvlogs_index_* pattern. You can modify the API as per your index name.
     In the following policy, the logs roll over daily or when the index size reaches 150 GB, whichever is earlier. Additionally, the logs are deleted after seven days, and you can only view logs up to seven days.

     "PUT _plugins/_ism/policies/cvltlogs-rollover-policy"{
     "policy":{
         "description":"Commvault logs rollover policy",
         "default_state":"hot",
         "schema_version":1,
         "states":[
         {
             "name":"hot",
             "actions":[
                 {
                 "rollover":{
                     "min_index_age":"1d",
                     "min_size":"150gb"
                 }
                 }
             ],
             "transitions":[
                 {
                 "state_name":"delete",
                 "conditions":{
                     "min_index_age":"7d"
                 }
                 }
             ]
         },
         {
             "name":"delete",
             "actions":[
                 {
                 "delete":{
     
                 }
                 }
             ]
         }
         ],
         "ism_template":{
         "index_patterns":[
             "cvlogs_index_*"
         ],
         "priority":100
         }
     }
     }

Results

The logs for the selected clients and client groups appear on the OpenSearch dashboard.