Hello Everyone,
I am currently using Elastic Search Version 8.8.1 installed on RHEL os.
Filebeat Version: 8.6.1
The logs are read from the Application server and pushed to Elasticsearch index using filebeat.
Data structure in each log file:
[2023-06-22 06:18:26.6566218] [ 24] [DEBUG] [CNP.AMS.Common.Utility.LogScope] - [Finished [OracleDBHelper.ExecuteQuery] in 10 ms]
The following fields are read and added to the payload:
Timestamp
LogthreadID
LogLevel
ClassName
LogMessage
Filebeat Input Type is Set as Logs
Dissector from Filebeat.yml
processors:
Following pattern dissects the message into fields and add it to the main payload.
- dissect:
tokenizer: '[%{TimeStamp}] [%{ThreadId}] [%{LogLevel}] [%{ClassName}] %{LogMessage}'
field: "message"
target_prefix: ""
ignore_failure: true
trim_values: left
Following pattern replaces any with a empty string
- replace:
fields:
- field: "LogMessage"
pattern: '['
replacement: ""
- field: "LogMessage"
pattern: ']'
replacement: ""
- field: "LogMessage"
pattern: '-'
replacement: ""
ignore_missing: true
fail_on_error: false
Following pattern converts the log creation timestamp from text to datetime datatype
- timestamp:
field: TimeStamp
timezone: America/Chicago
ignore_missing: true
ignore_failure: true
layouts:
- '2006-01-02T15:04:05Z'
- '2006-01-02T15:04:05.999'
- '2006-01-02T15:04:05.999Z'
- '2006-01-02T15:04:05.000000'
- '2006-01-02 15:04:05.0000000'
test:
- '2023-02-18T16:23:24.00Z'
- '2023-02-18T16:23:24.000'
- '2023-02-18T16:23:24.000Z'
- '2023-02-18T16:23:24.000563'
- '2023-02-18 16:23:24.0005633'
Issue:
I would like to apply a regex or any string function on the field LogMessage and look for a specific format , extract the string and create a new field and add it to the payload.
Ex:
Log Message: Finished OracleDBHelper.ExecuteQuery in 47 ms
Extract String: OracleDBHelper.ExecuteQuery
Create a New Field: LogMethod
Add the string to the new field. If the regex fails, I want to add a default string as "None"
What processor can be used for string extractions from a existing field using regex. Please let me know. Any help or suggestions greatly appreciated.
Thanks