How to extract string from the log and create a new field and send to elastic search index

Elastic Stack Beats
1

Hello Everyone,
I am currently using Elastic Search Version 8.8.1 installed on RHEL os.
Filebeat Version: 8.6.1

The logs are read from the Application server and pushed to Elasticsearch index using filebeat.
Data structure in each log file:
[2023-06-22 06:18:26.6566218] [ 24] [DEBUG] [CNP.AMS.Common.Utility.LogScope] - [Finished [OracleDBHelper.ExecuteQuery] in 10 ms]

The following fields are read and added to the payload:
Timestamp
LogthreadID
LogLevel
ClassName
LogMessage

Filebeat Input Type is Set as Logs

Dissector from Filebeat.yml
processors:

Following pattern dissects the message into fields and add it to the main payload.

  • dissect:
    tokenizer: '[%{TimeStamp}] [%{ThreadId}] [%{LogLevel}] [%{ClassName}] %{LogMessage}'
    field: "message"
    target_prefix: ""
    ignore_failure: true
    trim_values: left

Following pattern replaces any with a empty string

  • replace:
    fields:
    - field: "LogMessage"
    pattern: '['
    replacement: ""
    - field: "LogMessage"
    pattern: ']'
    replacement: ""
    - field: "LogMessage"
    pattern: '-'
    replacement: ""
    ignore_missing: true
    fail_on_error: false

Following pattern converts the log creation timestamp from text to datetime datatype

  • timestamp:
    field: TimeStamp
    timezone: America/Chicago
    ignore_missing: true
    ignore_failure: true
    layouts:
    - '2006-01-02T15:04:05Z'
    - '2006-01-02T15:04:05.999'
    - '2006-01-02T15:04:05.999Z'
    - '2006-01-02T15:04:05.000000'
    - '2006-01-02 15:04:05.0000000'
    test:
    - '2023-02-18T16:23:24.00Z'
    - '2023-02-18T16:23:24.000'
    - '2023-02-18T16:23:24.000Z'
    - '2023-02-18T16:23:24.000563'
    - '2023-02-18 16:23:24.0005633'

Issue:
I would like to apply a regex or any string function on the field LogMessage and look for a specific format , extract the string and create a new field and add it to the payload.

Ex:
Log Message: Finished OracleDBHelper.ExecuteQuery in 47 ms
Extract String: OracleDBHelper.ExecuteQuery
Create a New Field: LogMethod
Add the string to the new field. If the regex fails, I want to add a default string as "None"

What processor can be used for string extractions from a existing field using regex. Please let me know. Any help or suggestions greatly appreciated.

Thanks

2

You can use something like this:

input {
  generator {
       message => "[2023-06-12 06:18:26.6566218] [ 24] [DEBUG] [CNP.AMS.Common.Utility.LogScope] - [Finished [OracleDBHelper.ExecuteQuery] in 10 ms]"
       count => 1
  }
 
} # input

filter {
 dissect {
	mapping => {
        "message" => "[%{TimeStamp}] [%{ThreadId}] [%{LogLevel}] [%{ClassName}] - [%{LogMessage}]%{+LogMessage}]"
	}
 }
            grok { 
                match => { "LogMessage" => "%{WORD:status}%{SPACE}\[%{DATA:method}\]%{DATA}%{POSINT:exectime}%{SPACE}%{WORD:unit}" }
            }
      mutate {
         strip => ["ThreadId"]
      }
      date {
        match => ["TimeStamp", "yyyy-MM-dd HH:mm:ss.SSSSSSS"]
        timezone => "America/Chicago" 
        target=> "TimeStamp" 
      }
   
}

output {
    stdout { codec => rubydebug{}}
}

Result

{
       "message" => "[2023-06-12 06:18:26.6566218] [ 24] [DEBUG] [CNP.AMS.Common.Utility.LogScope] - [Finished [OracleDBHelper.ExecuteQuery] in 10 ms]",
          "unit" => "ms",
    "@timestamp" => 2023-06-23T22:44:08.412404800Z,
      "LogLevel" => "DEBUG",
     "TimeStamp" => 2023-06-12T11:18:26.656Z,
      "ThreadId" => "24",
    "LogMessage" => "Finished [OracleDBHelper.ExecuteQuery] in 10 ms",
      "exectime" => "10",
     "ClassName" => "CNP.AMS.Common.Utility.LogScope",
        "method" => "OracleDBHelper.ExecuteQuery",
        "status" => "Finished"
}
3

Hi Rios,
thanks for the update. I will try the Grok Processor on the Filebeat config and see if I could extract the string from the log message based on the inputs you have given.

Thanks once again, really appreciate your help.

Sarath