Hello,
First of all, I'm a new to filebeat, so I may say stupids things. Forgive me in advance.
I have to parse a log file that looks like :
2021-03-18 09:33:37,131 -- TYPE -- {"json1": "data", "json2": "data", "json3": "data"}
I would like to decode :
- Timestamp
- Type
- Json data
I tried to add the paring in the filebeat.inputs of the filebeat.yml file
processors:
- dissect:
tokenizer: "%{timestamp} -- %{type} -- %{json}"
field: "message"
target_prefix: "ocr.response"
- decode_json_fields:
fields: ["ocr.response.json"]
process_array: true
max_depth: 1
overwrite_keys: false
target: "json"
But I don't succeed in replacing the timestamp in Kibana by the read timestamp in my log file.
Then I tried to write a module
{
"description": "Pipeline for parsing ocr response logs",
"processors": [
{
"grok": {
"field": "message",
"patterns": [
"%{TIMESTAMP_ISO8601:ocr.response.timestamp} -- %{WORD:ocr.response.type} -- %{DATA:ocr.response.json}"
],
"pattern_definitions": {
"RESPONSE_TIME": "%{DAY} %{MONTH} %{MONTHDAY} %{TIME} %{YEAR}"
},
"ignore_missing": true
}
},
{
"json": {
"field": "ocr.response.json",
"target_field": "ocr.response.json_decoded"
}
},
{
"remove":{
"field": "message"
}
},
{
"rename": {
"field": "ocr.response.message1",
"target_field": "ocr.response.message",
"ignore_failure": true
}
},
{
"date": {
"field": "ocr.response.timestamp",
"target_field": "@timestamp",
"formats": ["EEE MMM dd H:m:s yyyy", "EEE MMM dd H:m:s.SSSSSS yyyy"],
"ignore_failure": true
}
},
{
"remove": {
"field": "ocr.response.timestamp",
"ignore_failure": true
}
}
],
"on_failure" : [{
"set" : {
"field" : "response.message",
"value" : "{{ _ingest.on_failure_message }}"
}
}]
}
Now, it's the json part that fails to be decoded.
Which is the best method for parsing a log file ?
What I am doing wrong ?
Regards,
Olivier