How to extract key-value pairs automatically from mixed type of message

Elastic Stack Logstash
1

hi
We have a mixed set of message coming from filebeat. The message is some form of "YAML/json" mix

[WARN ] 2020-09-02 16:03:55.966 [[pipline_id]>worker6] elasticsearch - Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"my_index", :routing=>nil, :_type=>"_doc"}, #<LogStash::Event:0x29cf370e>], :response=>{"index"=>{"_index"=>"my_index", "_type"=>"_doc", "_id"=>"qB6PT3QBbVYdMfr7deF1", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"object mapping for [error] tried to parse field [error] as object, but found a concrete value"}}}}

How to extract them into a hierarchy key-value basis automatically, without physically doing regex for each field?

So i'm looking for above message to be parsed and key-value's extracted as

message.status: 400
message.action.index._index: my_index
message.error.type: mapper_parsing_exception
...

Any idea how to do it?

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.