How to extract string from the log and create a new field and send to elastic search index

Hello Everyone,
I am currently using Elastic Search Version 8.8.1 installed on RHEL os.
Filebeat Version: 8.6.1

The logs are read from the Application server and pushed to Elasticsearch index using filebeat.
Data structure in each log file:
[2023-06-22 06:18:26.6566218] [ 24] [DEBUG] [CNP.AMS.Common.Utility.LogScope] - [Finished [OracleDBHelper.ExecuteQuery] in 10 ms]

The following fields are read and added to the payload:
Timestamp
LogthreadID
LogLevel
ClassName
LogMessage

Filebeat Input Type is Set as Logs

Dissector from Filebeat.yml
processors:

Following pattern dissects the message into fields and add it to the main payload.

  • dissect:
    tokenizer: '[%{TimeStamp}] [%{ThreadId}] [%{LogLevel}] [%{ClassName}] %{LogMessage}'
    field: "message"
    target_prefix: ""
    ignore_failure: true
    trim_values: left

Following pattern replaces any with a empty string

  • replace:
    fields:
    - field: "LogMessage"
    pattern: '['
    replacement: ""
    - field: "LogMessage"
    pattern: ']'
    replacement: ""
    - field: "LogMessage"
    pattern: '-'
    replacement: ""
    ignore_missing: true
    fail_on_error: false

Following pattern converts the log creation timestamp from text to datetime datatype

  • timestamp:
    field: TimeStamp
    timezone: America/Chicago
    ignore_missing: true
    ignore_failure: true
    layouts:
    - '2006-01-02T15:04:05Z'
    - '2006-01-02T15:04:05.999'
    - '2006-01-02T15:04:05.999Z'
    - '2006-01-02T15:04:05.000000'
    - '2006-01-02 15:04:05.0000000'
    test:
    - '2023-02-18T16:23:24.00Z'
    - '2023-02-18T16:23:24.000'
    - '2023-02-18T16:23:24.000Z'
    - '2023-02-18T16:23:24.000563'
    - '2023-02-18 16:23:24.0005633'

Issue:
I would like to apply a regex or any string function on the field LogMessage and look for a specific format , extract the string and create a new field and add it to the payload.

Ex:
Log Message: Finished OracleDBHelper.ExecuteQuery in 47 ms
Extract String: OracleDBHelper.ExecuteQuery
Create a New Field: LogMethod
Add the string to the new field. If the regex fails, I want to add a default string as "None"

What processor can be used for string extractions from a existing field using regex. Please let me know. Any help or suggestions greatly appreciated.

Thanks

You can use something like this:

input {
  generator {
       message => "[2023-06-12 06:18:26.6566218] [ 24] [DEBUG] [CNP.AMS.Common.Utility.LogScope] - [Finished [OracleDBHelper.ExecuteQuery] in 10 ms]"
       count => 1
  }
 
} # input

filter {
 dissect {
	mapping => {
        "message" => "[%{TimeStamp}] [%{ThreadId}] [%{LogLevel}] [%{ClassName}] - [%{LogMessage}]%{+LogMessage}]"
	}
 }
            grok { 
                match => { "LogMessage" => "%{WORD:status}%{SPACE}\[%{DATA:method}\]%{DATA}%{POSINT:exectime}%{SPACE}%{WORD:unit}" }
            }
      mutate {
         strip => ["ThreadId"]
      }
      date {
        match => ["TimeStamp", "yyyy-MM-dd HH:mm:ss.SSSSSSS"]
        timezone => "America/Chicago" 
        target=> "TimeStamp" 
      }
   
}

output {
    stdout { codec => rubydebug{}}
}

Result

{
       "message" => "[2023-06-12 06:18:26.6566218] [ 24] [DEBUG] [CNP.AMS.Common.Utility.LogScope] - [Finished [OracleDBHelper.ExecuteQuery] in 10 ms]",
          "unit" => "ms",
    "@timestamp" => 2023-06-23T22:44:08.412404800Z,
      "LogLevel" => "DEBUG",
     "TimeStamp" => 2023-06-12T11:18:26.656Z,
      "ThreadId" => "24",
    "LogMessage" => "Finished [OracleDBHelper.ExecuteQuery] in 10 ms",
      "exectime" => "10",
     "ClassName" => "CNP.AMS.Common.Utility.LogScope",
        "method" => "OracleDBHelper.ExecuteQuery",
        "status" => "Finished"
}
1 Like

Hi Rios,
thanks for the update. I will try the Grok Processor on the Filebeat config and see if I could extract the string from the log message based on the inputs you have given.

Thanks once again, really appreciate your help.

Sarath

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.