Process logs in json with a "message" key not working with filebeat 6.4.1

Hello,

I have an app that produce logs in that form :
{
"time": "2018-10-01T16:59:33+02:00",
"level": "INFO",
"component_name": "test",
"request_type": "incomming_request",
"request_peer": "front",
"session_id": "todo",
"request_id": "todo",
"request_method": "POST",
"request_uri": "/setconfig",
"request_status_code": "200",
"message": {
"remote_ip": "1.1.1.1",
"timeout": "10"
}
}

My filebeat config :
- paths:
- "output.log"
json.keys_under_root: true
json.add_error_key: true
json.overwrite_keys: true
type: log

I am aware that "message" key is used in the Elastic Common Schema thats why i use the "json.overwrite_keys: true" option.

But i still have theses warnings in filebeat logs {"type":"mapper_parsing_exception","reason":"failed to parse [message]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:396"} and the events are not sent to elasticsearch.

Is that a normal behaviour ? With the overwrite key set to true this shouldnt be a problem ?

If you modify the type of an existing field, you need to update your index mapping in Elasticsearch, because it still expects message field to be a string. Thus, indexing fails.

I dont use custom maping, i was thinking filebeat was sending one to ES with the right types acording to his configuration (in my case json object)

So for my case is better to go with a custom template and specify the type of message ?

Exactly. Unfortunately, Filebeat comes with a default generated mapping based on the official exported fields. It does not take into account the possible modifications by users.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.