How to extract string from the log and create a new field and send to elastic search index

[mbsarathchandra (Sarath Chandra)]

Hello Everyone,
I am currently using Elastic Search Version 8.8.1 installed on RHEL os.
Filebeat Version: 8.6.1

The logs are read from the Application server and pushed to Elasticsearch index using filebeat.
Data structure in each log file:
[2023-06-22 06:18:26.6566218] [ 24] [DEBUG] [CNP.AMS.Common.Utility.LogScope] - [Finished [OracleDBHelper.ExecuteQuery] in 10 ms]

The following fields are read and added to the payload:
Timestamp
LogthreadID
LogLevel
ClassName
LogMessage

Filebeat Input Type is Set as Logs

Dissector from Filebeat.yml
processors:

Following pattern dissects the message into fields and add it to the main payload.

• dissect:
  tokenizer: '[%{TimeStamp}] [%{ThreadId}] [%{LogLevel}] [%{ClassName}] %{LogMessage}'
  field: "message"
  target_prefix: ""
  ignore_failure: true
  trim_values: left

Following pattern replaces any with a empty string

• replace:
  fields:
  - field: "LogMessage"
  pattern: '['
  replacement: ""
  - field: "LogMessage"
  pattern: ']'
  replacement: ""
  - field: "LogMessage"
  pattern: '-'
  replacement: ""
  ignore_missing: true
  fail_on_error: false

Following pattern converts the log creation timestamp from text to datetime datatype

• timestamp:
  field: TimeStamp
  timezone: America/Chicago
  ignore_missing: true
  ignore_failure: true
  layouts:
  - '2006-01-02T15:04:05Z'
  - '2006-01-02T15:04:05.999'
  - '2006-01-02T15:04:05.999Z'
  - '2006-01-02T15:04:05.000000'
  - '2006-01-02 15:04:05.0000000'
  test:
  - '2023-02-18T16:23:24.00Z'
  - '2023-02-18T16:23:24.000'
  - '2023-02-18T16:23:24.000Z'
  - '2023-02-18T16:23:24.000563'
  - '2023-02-18 16:23:24.0005633'

Issue:
I would like to apply a regex or any string function on the field LogMessage and look for a specific format , extract the string and create a new field and add it to the payload.

Ex:
Log Message: Finished OracleDBHelper.ExecuteQuery in 47 ms
Extract String: OracleDBHelper.ExecuteQuery
Create a New Field: LogMethod
Add the string to the new field. If the regex fails, I want to add a default string as "None"

What processor can be used for string extractions from a existing field using regex. Please let me know. Any help or suggestions greatly appreciated.

Thanks

[Rios]

You can use something like this:

input {
  generator {
       message => "[2023-06-12 06:18:26.6566218] [ 24] [DEBUG] [CNP.AMS.Common.Utility.LogScope] - [Finished [OracleDBHelper.ExecuteQuery] in 10 ms]"
       count => 1
  }
 
} # input

filter {
 dissect {
	mapping => {
        "message" => "[%{TimeStamp}] [%{ThreadId}] [%{LogLevel}] [%{ClassName}] - [%{LogMessage}]%{+LogMessage}]"
	}
 }
            grok { 
                match => { "LogMessage" => "%{WORD:status}%{SPACE}\[%{DATA:method}\]%{DATA}%{POSINT:exectime}%{SPACE}%{WORD:unit}" }
            }
      mutate {
         strip => ["ThreadId"]
      }
      date {
        match => ["TimeStamp", "yyyy-MM-dd HH:mm:ss.SSSSSSS"]
        timezone => "America/Chicago" 
        target=> "TimeStamp" 
      }
   
}

output {
    stdout { codec => rubydebug{}}
}

Result

{
       "message" => "[2023-06-12 06:18:26.6566218] [ 24] [DEBUG] [CNP.AMS.Common.Utility.LogScope] - [Finished [OracleDBHelper.ExecuteQuery] in 10 ms]",
          "unit" => "ms",
    "@timestamp" => 2023-06-23T22:44:08.412404800Z,
      "LogLevel" => "DEBUG",
     "TimeStamp" => 2023-06-12T11:18:26.656Z,
      "ThreadId" => "24",
    "LogMessage" => "Finished [OracleDBHelper.ExecuteQuery] in 10 ms",
      "exectime" => "10",
     "ClassName" => "CNP.AMS.Common.Utility.LogScope",
        "method" => "OracleDBHelper.ExecuteQuery",
        "status" => "Finished"
}

[mbsarathchandra (Sarath Chandra)]

@Rios

OracleDBHelper.ExecuteQuery"

Hi Rios,
thanks for the update. I will try the Grok Processor on the Filebeat config and see if I could extract the string from the log message based on the inputs you have given.

Thanks once again, really appreciate your help.

Sarath